Nov 03 01:50:45.997462 ip-10-0-137-69 systemd[1]: Starting Kubernetes Kubelet... Nov 03 01:50:46.390754 ip-10-0-137-69 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:50:46.390754 ip-10-0-137-69 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 01:50:46.390754 ip-10-0-137-69 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:50:46.390754 ip-10-0-137-69 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:50:46.393927 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393818 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:50:46.393927 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393927 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:50:46.393992 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393932 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:50:46.393992 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393935 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:50:46.393992 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393938 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:50:46.393992 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393941 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:50:46.393992 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393944 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:50:46.393992 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393947 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:50:46.393992 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393949 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:50:46.393992 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393952 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:50:46.393992 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393955 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:50:46.393992 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393959 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:50:46.393992 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393963 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:50:46.393992 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393966 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:50:46.393992 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393969 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:50:46.393992 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393971 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:50:46.393992 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393974 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:50:46.393992 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393977 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:50:46.393992 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393979 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:50:46.393992 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393982 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:50:46.393992 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393984 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:50:46.394451 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393987 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:50:46.394451 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393993 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:50:46.394451 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393996 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:50:46.394451 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.393999 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:50:46.394451 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394002 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:50:46.394451 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394005 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:50:46.394451 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394008 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:50:46.394451 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394010 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:50:46.394451 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394013 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:50:46.394451 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394015 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:50:46.394451 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394018 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:50:46.394451 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394020 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:50:46.394451 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394023 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:50:46.394451 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394026 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:50:46.394451 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394028 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:50:46.394451 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394031 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:50:46.394451 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394033 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:50:46.394451 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394035 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:50:46.394451 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394038 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:50:46.394451 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394040 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:50:46.394937 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394043 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:50:46.394937 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394045 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:50:46.394937 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394048 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:50:46.394937 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394051 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:50:46.394937 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394053 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:50:46.394937 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394056 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:50:46.394937 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394058 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:50:46.394937 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394063 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:50:46.394937 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394065 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:50:46.394937 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394068 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:50:46.394937 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394070 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:50:46.394937 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394072 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:50:46.394937 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394077 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:50:46.394937 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394081 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:50:46.394937 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394084 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:50:46.394937 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394086 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:50:46.394937 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394089 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:50:46.394937 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394092 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:50:46.394937 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394095 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:50:46.395393 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394097 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:50:46.395393 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394100 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:50:46.395393 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394102 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:50:46.395393 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394105 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:50:46.395393 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394107 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:50:46.395393 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394110 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:50:46.395393 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394112 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:50:46.395393 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394115 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:50:46.395393 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394117 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:50:46.395393 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394120 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:50:46.395393 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394122 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:50:46.395393 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394125 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:50:46.395393 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394128 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:50:46.395393 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394130 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:50:46.395393 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394133 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:50:46.395393 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394135 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:50:46.395393 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394138 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:50:46.395393 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394140 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:50:46.395393 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394143 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:50:46.395393 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394146 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:50:46.395886 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394148 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:50:46.395886 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394151 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:50:46.395886 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394154 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:50:46.395886 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394156 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:50:46.395886 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394159 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:50:46.395886 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394161 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:50:46.395886 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394164 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:50:46.395886 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394167 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:50:46.395886 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394170 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:50:46.395886 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394173 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:50:46.395886 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394175 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:50:46.395886 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394178 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:50:46.395886 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394181 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:50:46.395886 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394183 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:50:46.395886 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394186 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:50:46.395886 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394200 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:50:46.395886 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394203 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:50:46.395886 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394206 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:50:46.395886 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394209 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:50:46.395886 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394211 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:50:46.396420 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394214 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:50:46.396420 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394217 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:50:46.396420 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394220 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:50:46.396420 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394222 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:50:46.396420 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394225 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:50:46.396420 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.394227 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:50:46.396420 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395012 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:50:46.396420 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395019 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:50:46.396420 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395022 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:50:46.396420 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395024 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:50:46.396420 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395027 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:50:46.396420 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395030 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:50:46.396420 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395032 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:50:46.396420 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395035 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:50:46.396420 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395039 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:50:46.396420 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395041 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:50:46.396420 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395044 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:50:46.396420 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395047 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:50:46.396420 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395052 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:50:46.396911 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395057 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:50:46.396911 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395061 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:50:46.396911 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395064 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:50:46.396911 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395067 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:50:46.396911 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395071 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:50:46.396911 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395075 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:50:46.396911 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395078 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:50:46.396911 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395080 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:50:46.396911 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395083 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:50:46.396911 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395086 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:50:46.396911 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395088 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:50:46.396911 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395091 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:50:46.396911 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395093 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:50:46.396911 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395096 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:50:46.396911 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395098 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:50:46.396911 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395101 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:50:46.396911 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395103 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:50:46.396911 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395106 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:50:46.396911 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395108 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:50:46.396911 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395111 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:50:46.397395 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395113 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:50:46.397395 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395116 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:50:46.397395 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395118 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:50:46.397395 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395121 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:50:46.397395 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395123 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:50:46.397395 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395126 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:50:46.397395 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395129 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:50:46.397395 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395132 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:50:46.397395 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395136 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:50:46.397395 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395138 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:50:46.397395 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395141 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:50:46.397395 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395144 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:50:46.397395 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395147 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:50:46.397395 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395149 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:50:46.397395 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395152 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:50:46.397395 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395154 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:50:46.397395 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395157 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:50:46.397395 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395159 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:50:46.397395 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395162 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:50:46.397395 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395164 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:50:46.397900 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395166 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:50:46.397900 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395169 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:50:46.397900 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395171 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:50:46.397900 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395174 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:50:46.397900 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395177 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:50:46.397900 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395179 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:50:46.397900 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395182 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:50:46.397900 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395185 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:50:46.397900 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395200 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:50:46.397900 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395203 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:50:46.397900 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395206 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:50:46.397900 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395208 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:50:46.397900 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395211 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:50:46.397900 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395214 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:50:46.397900 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395217 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:50:46.397900 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395220 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:50:46.397900 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395222 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:50:46.397900 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395225 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:50:46.397900 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395228 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:50:46.398363 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395230 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:50:46.398363 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395233 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:50:46.398363 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395236 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:50:46.398363 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395239 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:50:46.398363 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395242 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:50:46.398363 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395245 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:50:46.398363 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395248 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:50:46.398363 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395250 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:50:46.398363 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395253 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:50:46.398363 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395255 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:50:46.398363 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395258 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:50:46.398363 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395260 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:50:46.398363 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395263 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:50:46.398363 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395265 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:50:46.398363 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395268 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:50:46.398363 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395270 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:50:46.398363 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395273 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:50:46.398363 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395275 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:50:46.398363 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395278 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:50:46.398821 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395280 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:50:46.398821 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395283 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:50:46.398821 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395285 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:50:46.398821 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395288 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:50:46.398821 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395290 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:50:46.398821 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395293 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:50:46.398821 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395295 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:50:46.398821 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395298 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:50:46.398821 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395300 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:50:46.398821 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395303 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:50:46.398821 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395305 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:50:46.398821 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395307 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:50:46.398821 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395310 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:50:46.398821 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395313 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:50:46.398821 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395316 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:50:46.398821 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395803 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:50:46.398821 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395808 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:50:46.398821 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395811 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:50:46.398821 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395814 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:50:46.399303 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395816 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:50:46.399303 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395819 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:50:46.399303 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395822 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:50:46.399303 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395824 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:50:46.399303 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395827 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:50:46.399303 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395830 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:50:46.399303 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395832 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:50:46.399303 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395835 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:50:46.399303 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395838 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:50:46.399303 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395841 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:50:46.399303 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395844 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:50:46.399303 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395846 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:50:46.399303 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395849 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:50:46.399303 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395851 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:50:46.399303 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395854 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:50:46.399303 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395856 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:50:46.399303 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395859 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:50:46.399303 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395862 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:50:46.399303 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395864 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:50:46.399764 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395867 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:50:46.399764 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395870 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:50:46.399764 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395872 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:50:46.399764 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395875 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:50:46.399764 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395877 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:50:46.399764 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395882 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:50:46.399764 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395885 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:50:46.399764 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395888 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:50:46.399764 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395891 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:50:46.399764 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395893 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:50:46.399764 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395897 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:50:46.399764 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395900 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:50:46.399764 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395903 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:50:46.399764 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395905 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:50:46.399764 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395908 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:50:46.399764 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395910 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:50:46.399764 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395913 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:50:46.399764 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395915 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:50:46.399764 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395918 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:50:46.400241 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395920 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:50:46.400241 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395923 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:50:46.400241 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395925 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:50:46.400241 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395928 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:50:46.400241 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395930 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:50:46.400241 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395932 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:50:46.400241 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395935 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:50:46.400241 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395937 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:50:46.400241 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395939 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:50:46.400241 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395942 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:50:46.400241 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395944 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:50:46.400241 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395947 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:50:46.400241 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395949 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:50:46.400241 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395954 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:50:46.400241 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395957 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:50:46.400241 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395960 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:50:46.400241 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395962 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:50:46.400241 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395965 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:50:46.400241 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395968 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:50:46.400241 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395970 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:50:46.400726 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395973 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:50:46.400726 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395976 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:50:46.400726 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395978 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:50:46.400726 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395981 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:50:46.400726 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395984 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:50:46.400726 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395986 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:50:46.400726 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395989 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:50:46.400726 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395991 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:50:46.400726 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395994 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:50:46.400726 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395996 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:50:46.400726 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.395999 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:50:46.400726 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396001 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:50:46.400726 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396004 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:50:46.400726 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396006 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:50:46.400726 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396008 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:50:46.400726 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396011 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:50:46.400726 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396013 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:50:46.400726 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396016 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:50:46.400726 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396020 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:50:46.400726 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396023 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:50:46.401207 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396025 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:50:46.401207 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396028 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:50:46.401207 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396030 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:50:46.401207 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396033 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:50:46.401207 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396035 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:50:46.401207 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396040 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:50:46.401207 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396043 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:50:46.401207 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396045 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:50:46.401207 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396048 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:50:46.401207 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396050 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:50:46.401207 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396053 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:50:46.401207 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396055 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:50:46.401207 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396058 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:50:46.401207 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396063 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:50:46.401207 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396067 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:50:46.401207 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396070 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:50:46.401207 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396073 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:50:46.401207 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396076 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:50:46.401207 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396079 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396083 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396085 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396088 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396090 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.396093 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397303 2577 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397313 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397323 2577 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397328 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397333 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397337 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397343 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397347 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397351 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397354 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397357 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397361 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397364 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397367 2577 flags.go:64] FLAG: --cgroup-root="" Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397370 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397373 2577 flags.go:64] FLAG: --client-ca-file="" Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397376 2577 flags.go:64] FLAG: --cloud-provider="external" Nov 03 01:50:46.401668 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397379 2577 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397384 2577 flags.go:64] FLAG: --cluster-domain="" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397387 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397390 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397393 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397397 2577 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397401 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397404 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397407 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397411 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397415 2577 flags.go:64] FLAG: --contention-profiling="false" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397418 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397420 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397424 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397427 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397432 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397435 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397438 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397440 2577 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397444 2577 flags.go:64] FLAG: --enable-server="true" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397446 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397452 2577 flags.go:64] FLAG: --event-burst="100" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397455 2577 flags.go:64] FLAG: --event-qps="50" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397458 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 01:50:46.402214 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397461 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397464 2577 flags.go:64] FLAG: --eviction-hard="" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397468 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397471 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397475 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397478 2577 flags.go:64] FLAG: --eviction-soft="" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397481 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397484 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397487 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397490 2577 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397493 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397496 2577 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397499 2577 flags.go:64] FLAG: --feature-gates="" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397503 2577 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397506 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397509 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397512 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397515 2577 flags.go:64] FLAG: --healthz-port="10248" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397519 2577 flags.go:64] FLAG: --help="false" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397522 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397525 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397528 2577 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397531 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 01:50:46.402829 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397535 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397538 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397541 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397544 2577 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397547 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397550 2577 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397555 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397558 2577 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397561 2577 flags.go:64] FLAG: --kube-reserved="" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397564 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397567 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397570 2577 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397573 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397576 2577 flags.go:64] FLAG: --lock-file="" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397579 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397582 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397585 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397591 2577 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397594 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397597 2577 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397600 2577 flags.go:64] FLAG: --logging-format="text" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397602 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397606 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397609 2577 flags.go:64] FLAG: --manifest-url="" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397612 2577 flags.go:64] FLAG: --manifest-url-header="" Nov 03 01:50:46.403409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397616 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397619 2577 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397623 2577 flags.go:64] FLAG: --max-pods="110" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397626 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397630 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397633 2577 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397635 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397638 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397641 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397644 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397652 2577 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397655 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397658 2577 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397663 2577 flags.go:64] FLAG: --pod-cidr="" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397666 2577 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397669 2577 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397672 2577 flags.go:64] FLAG: --pods-per-core="0" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397675 2577 flags.go:64] FLAG: --port="10250" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397677 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397680 2577 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-069bb454c0d19fd90" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397684 2577 flags.go:64] FLAG: --qos-reserved="" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397687 2577 flags.go:64] FLAG: --read-only-port="10255" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397690 2577 flags.go:64] FLAG: --register-node="true" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397693 2577 flags.go:64] FLAG: --register-with-taints="" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397696 2577 flags.go:64] FLAG: --registry-burst="10" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397700 2577 flags.go:64] FLAG: --registry-qps="5" Nov 03 01:50:46.404048 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397702 2577 flags.go:64] FLAG: --reserved-cpus="" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397705 2577 flags.go:64] FLAG: --reserved-memory="" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397709 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397712 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397715 2577 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397718 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397721 2577 flags.go:64] FLAG: --runonce="false" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397724 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397727 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397730 2577 flags.go:64] FLAG: --seccomp-default="false" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397734 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397736 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397739 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397742 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397745 2577 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397748 2577 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397751 2577 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397754 2577 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397757 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397761 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397764 2577 flags.go:64] FLAG: --system-cgroups="" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397767 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397772 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397775 2577 flags.go:64] FLAG: --tls-cert-file="" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397778 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 01:50:46.404678 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397782 2577 flags.go:64] FLAG: --tls-min-version="" Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397787 2577 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397790 2577 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397793 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397796 2577 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397799 2577 flags.go:64] FLAG: --v="2" Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397804 2577 flags.go:64] FLAG: --version="false" Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397809 2577 flags.go:64] FLAG: --vmodule="" Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397813 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.397816 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.397946 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.397950 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.397954 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.397957 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.397959 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.397962 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.397965 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.397967 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.397971 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.397973 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.397976 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.397979 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:50:46.405345 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.397981 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:50:46.405866 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.397984 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:50:46.405866 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.397986 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:50:46.405866 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.397989 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:50:46.405866 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.397993 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:50:46.405866 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.397996 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:50:46.405866 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.397998 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:50:46.405866 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398002 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:50:46.405866 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398007 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:50:46.405866 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398010 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:50:46.405866 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398013 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:50:46.405866 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398017 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:50:46.405866 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398021 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:50:46.405866 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398023 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:50:46.405866 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398026 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:50:46.405866 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398029 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:50:46.405866 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398031 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:50:46.405866 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398034 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:50:46.405866 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398036 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:50:46.405866 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398039 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:50:46.406341 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398041 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:50:46.406341 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398044 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:50:46.406341 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398046 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:50:46.406341 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398049 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:50:46.406341 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398052 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:50:46.406341 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398055 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:50:46.406341 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398057 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:50:46.406341 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398060 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:50:46.406341 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398063 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:50:46.406341 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398066 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:50:46.406341 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398069 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:50:46.406341 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398071 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:50:46.406341 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398074 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:50:46.406341 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398076 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:50:46.406341 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398079 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:50:46.406341 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398081 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:50:46.406341 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398085 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:50:46.406341 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398087 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:50:46.406341 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398090 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:50:46.406826 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398092 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:50:46.406826 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398095 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:50:46.406826 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398098 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:50:46.406826 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398100 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:50:46.406826 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398104 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:50:46.406826 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398106 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:50:46.406826 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398109 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:50:46.406826 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398112 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:50:46.406826 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398114 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:50:46.406826 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398117 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:50:46.406826 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398119 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:50:46.406826 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398122 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:50:46.406826 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398125 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:50:46.406826 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398127 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:50:46.406826 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398130 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:50:46.406826 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398132 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:50:46.406826 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398136 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:50:46.406826 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398140 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:50:46.406826 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398143 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:50:46.407291 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398146 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:50:46.407291 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398149 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:50:46.407291 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398151 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:50:46.407291 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398154 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:50:46.407291 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398157 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:50:46.407291 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398159 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:50:46.407291 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398162 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:50:46.407291 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398164 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:50:46.407291 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398167 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:50:46.407291 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398169 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:50:46.407291 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398173 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:50:46.407291 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398176 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:50:46.407291 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398178 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:50:46.407291 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398181 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:50:46.407291 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398183 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:50:46.407291 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398186 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:50:46.407291 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398200 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:50:46.407291 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398203 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:50:46.407291 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398206 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:50:46.407291 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398209 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:50:46.407759 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398211 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:50:46.407759 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398214 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:50:46.407759 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398217 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:50:46.407759 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398219 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:50:46.407759 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398222 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:50:46.407759 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398225 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:50:46.407759 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398227 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:50:46.407759 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398230 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:50:46.407759 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398232 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:50:46.407759 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398235 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:50:46.407759 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398237 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:50:46.407759 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398240 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:50:46.407759 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398242 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:50:46.407759 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398245 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:50:46.407759 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398247 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:50:46.407759 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398250 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:50:46.408218 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.398277 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:50:46.408218 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398311 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:50:46.408218 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398316 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:50:46.408218 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398320 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:50:46.408218 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398323 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:50:46.408218 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398327 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:50:46.408218 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398333 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:50:46.408218 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398336 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:50:46.408218 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398339 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:50:46.408218 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398341 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:50:46.408218 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398344 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:50:46.408218 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398346 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:50:46.408218 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398349 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:50:46.408218 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398351 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:50:46.408218 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398355 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:50:46.408218 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398358 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:50:46.408608 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398361 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:50:46.408608 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398363 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:50:46.408608 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398366 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:50:46.408608 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398369 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:50:46.408608 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398371 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:50:46.408608 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398374 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:50:46.408608 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398377 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:50:46.408608 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398379 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:50:46.408608 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398382 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:50:46.408608 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398384 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:50:46.408608 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398387 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:50:46.408608 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398389 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:50:46.408608 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398392 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:50:46.408608 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398395 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:50:46.408608 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398397 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:50:46.408608 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398401 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:50:46.408608 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398404 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:50:46.408608 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398406 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:50:46.408608 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398409 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:50:46.408608 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398412 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:50:46.409076 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398415 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:50:46.409076 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398417 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:50:46.409076 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398421 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:50:46.409076 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398423 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:50:46.409076 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398426 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:50:46.409076 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398428 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:50:46.409076 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398430 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:50:46.409076 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398433 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:50:46.409076 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398435 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:50:46.409076 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398438 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:50:46.409076 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398442 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:50:46.409076 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398444 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:50:46.409076 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398447 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:50:46.409076 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398449 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:50:46.409076 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398452 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:50:46.409076 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398455 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:50:46.409076 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398458 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:50:46.409076 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398460 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:50:46.409076 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398463 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:50:46.409550 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398465 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:50:46.409550 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398468 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:50:46.409550 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398471 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:50:46.409550 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398473 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:50:46.409550 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398475 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:50:46.409550 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398478 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:50:46.409550 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398481 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:50:46.409550 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398483 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:50:46.409550 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398488 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:50:46.409550 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398491 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:50:46.409550 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398494 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:50:46.409550 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398497 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:50:46.409550 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398504 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:50:46.409550 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398507 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:50:46.409550 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398509 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:50:46.409550 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398512 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:50:46.409550 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398515 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:50:46.409550 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398517 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:50:46.409550 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398520 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:50:46.410226 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398522 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:50:46.410226 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398525 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:50:46.410226 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398527 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:50:46.410226 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398530 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:50:46.410226 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398534 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:50:46.410226 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398537 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:50:46.410226 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398540 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:50:46.410226 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398542 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:50:46.410226 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398545 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:50:46.410226 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398548 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:50:46.410226 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398550 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:50:46.410226 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398553 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:50:46.410226 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398555 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:50:46.410226 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398558 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:50:46.410226 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398561 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:50:46.410226 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398563 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:50:46.410226 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398566 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:50:46.410226 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398568 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:50:46.410226 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398571 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:50:46.410226 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398573 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:50:46.410954 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398576 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:50:46.410954 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398579 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:50:46.410954 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398581 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:50:46.410954 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398584 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:50:46.410954 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398587 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:50:46.410954 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398589 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:50:46.410954 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398592 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:50:46.410954 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398595 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:50:46.410954 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398598 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:50:46.410954 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398600 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:50:46.410954 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398604 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:50:46.410954 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398606 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:50:46.410954 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.398609 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:50:46.410954 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.405035 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 01:50:46.410954 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.405049 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 01:50:46.410954 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.405065 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 01:50:46.410954 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.405070 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 01:50:46.410954 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405126 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:50:46.410954 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405131 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:50:46.411431 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405134 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:50:46.411431 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405138 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:50:46.411431 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405140 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:50:46.411431 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405143 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:50:46.411431 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405145 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:50:46.411431 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405148 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:50:46.411431 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405150 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:50:46.411431 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405153 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:50:46.411431 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405156 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:50:46.411431 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405158 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:50:46.411431 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405161 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:50:46.411431 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405163 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:50:46.411431 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405166 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:50:46.411431 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405168 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:50:46.411431 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405171 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:50:46.411431 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405173 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:50:46.411431 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405176 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:50:46.411431 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405179 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:50:46.411431 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405181 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:50:46.411431 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405184 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:50:46.411948 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405186 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:50:46.411948 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405203 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:50:46.411948 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405207 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:50:46.411948 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405209 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:50:46.411948 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405212 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:50:46.411948 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405214 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:50:46.411948 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405217 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:50:46.411948 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405222 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:50:46.411948 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405226 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:50:46.411948 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405229 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:50:46.411948 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405232 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:50:46.411948 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405234 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:50:46.411948 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405237 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:50:46.411948 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405240 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:50:46.411948 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405242 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:50:46.411948 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405245 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:50:46.411948 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405247 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:50:46.411948 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405250 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:50:46.412409 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405254 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:50:46.412409 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405258 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:50:46.412409 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405260 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:50:46.412409 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405263 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:50:46.412409 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405265 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:50:46.412409 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405268 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:50:46.412409 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405271 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:50:46.412409 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405273 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:50:46.412409 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405276 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:50:46.412409 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405278 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:50:46.412409 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405281 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:50:46.412409 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405283 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:50:46.412409 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405286 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:50:46.412409 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405289 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:50:46.412409 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405291 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:50:46.412409 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405294 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:50:46.412409 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405297 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:50:46.412409 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405299 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:50:46.412409 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405302 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:50:46.412409 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405304 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:50:46.412912 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405307 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:50:46.412912 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405309 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:50:46.412912 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405312 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:50:46.412912 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405315 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:50:46.412912 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405317 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:50:46.412912 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405320 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:50:46.412912 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405323 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:50:46.412912 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405325 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:50:46.412912 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405327 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:50:46.412912 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405330 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:50:46.412912 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405332 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:50:46.412912 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405335 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:50:46.412912 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405337 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:50:46.412912 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405340 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:50:46.412912 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405342 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:50:46.412912 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405345 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:50:46.412912 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405347 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:50:46.412912 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405350 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:50:46.412912 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405353 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:50:46.413481 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405355 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:50:46.413481 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405358 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:50:46.413481 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405361 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:50:46.413481 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405363 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:50:46.413481 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405366 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:50:46.413481 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405368 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:50:46.413481 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405371 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:50:46.413481 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405373 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:50:46.413481 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405376 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:50:46.413481 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405378 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:50:46.413481 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405381 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:50:46.413481 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405383 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:50:46.413481 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405386 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:50:46.413481 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405388 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:50:46.413481 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405391 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:50:46.413481 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405393 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:50:46.413481 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405396 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:50:46.413481 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405398 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:50:46.413481 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405401 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:50:46.413481 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405403 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:50:46.414014 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405406 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:50:46.414014 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405408 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:50:46.414014 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405411 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:50:46.414014 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405413 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:50:46.414014 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405416 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:50:46.414014 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405418 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:50:46.414014 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405420 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:50:46.414014 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.405446 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:50:46.414014 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405593 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:50:46.414014 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405598 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:50:46.414014 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405600 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:50:46.414014 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405603 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:50:46.414014 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405607 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:50:46.414014 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405609 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:50:46.414014 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405612 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:50:46.414014 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405614 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:50:46.414404 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405617 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:50:46.414404 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405619 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:50:46.414404 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405622 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:50:46.414404 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405626 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:50:46.414404 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405629 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:50:46.414404 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405632 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:50:46.414404 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405634 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:50:46.414404 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405636 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:50:46.414404 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405639 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:50:46.414404 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405641 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:50:46.414404 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405644 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:50:46.414404 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405646 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:50:46.414404 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405649 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:50:46.414404 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405652 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:50:46.414404 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405654 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:50:46.414404 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405657 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:50:46.414404 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405659 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:50:46.414404 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405662 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:50:46.414404 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405664 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:50:46.414836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405668 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:50:46.414836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405671 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:50:46.414836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405674 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:50:46.414836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405676 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:50:46.414836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405679 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:50:46.414836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405681 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:50:46.414836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405684 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:50:46.414836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405686 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:50:46.414836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405689 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:50:46.414836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405692 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:50:46.414836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405694 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:50:46.414836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405697 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:50:46.414836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405699 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:50:46.414836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405702 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:50:46.414836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405704 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:50:46.414836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405709 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:50:46.414836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405712 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:50:46.414836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405715 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:50:46.414836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405717 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:50:46.414836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405720 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:50:46.415311 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405722 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:50:46.415311 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405725 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:50:46.415311 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405727 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:50:46.415311 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405730 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:50:46.415311 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405732 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:50:46.415311 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405735 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:50:46.415311 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405738 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:50:46.415311 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405740 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:50:46.415311 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405743 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:50:46.415311 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405746 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:50:46.415311 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405748 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:50:46.415311 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405751 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:50:46.415311 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405753 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:50:46.415311 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405756 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:50:46.415311 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405759 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:50:46.415311 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405761 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:50:46.415311 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405764 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:50:46.415311 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405766 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:50:46.415311 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405769 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:50:46.415763 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405772 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:50:46.415763 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405774 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:50:46.415763 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405777 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:50:46.415763 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405780 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:50:46.415763 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405783 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:50:46.415763 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405785 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:50:46.415763 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405788 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:50:46.415763 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405790 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:50:46.415763 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405794 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:50:46.415763 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405796 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:50:46.415763 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405799 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:50:46.415763 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405801 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:50:46.415763 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405804 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:50:46.415763 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405807 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:50:46.415763 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405809 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:50:46.415763 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405811 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:50:46.415763 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405814 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:50:46.415763 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405817 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:50:46.415763 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405819 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:50:46.416237 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405821 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:50:46.416237 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405824 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:50:46.416237 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405827 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:50:46.416237 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405830 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:50:46.416237 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405833 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:50:46.416237 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405837 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:50:46.416237 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405839 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:50:46.416237 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405842 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:50:46.416237 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405845 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:50:46.416237 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405847 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:50:46.416237 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405850 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:50:46.416237 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405852 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:50:46.416237 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405855 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:50:46.416237 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405857 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:50:46.416237 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405860 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:50:46.416237 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405863 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:50:46.416237 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405865 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:50:46.416237 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405868 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:50:46.416237 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405870 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:50:46.416237 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405873 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:50:46.416737 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405876 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:50:46.416737 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.405898 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:50:46.416737 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405929 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:50:46.416737 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405933 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:50:46.416737 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405936 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:50:46.416737 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405939 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:50:46.416737 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405941 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:50:46.416737 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405943 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:50:46.416737 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405946 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:50:46.416737 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405948 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:50:46.416737 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405951 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:50:46.416737 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405954 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:50:46.416737 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405956 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:50:46.416737 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405959 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:50:46.416737 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405961 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:50:46.417122 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405964 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:50:46.417122 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405967 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:50:46.417122 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405972 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:50:46.417122 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405974 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:50:46.417122 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405977 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:50:46.417122 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405979 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:50:46.417122 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405982 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:50:46.417122 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405984 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:50:46.417122 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405987 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:50:46.417122 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405989 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:50:46.417122 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405992 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:50:46.417122 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405994 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:50:46.417122 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405997 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:50:46.417122 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.405999 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:50:46.417122 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406001 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:50:46.417122 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406004 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:50:46.417122 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406006 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:50:46.417122 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406009 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:50:46.417122 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406011 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:50:46.417586 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406014 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:50:46.417586 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406017 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:50:46.417586 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406019 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:50:46.417586 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406022 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:50:46.417586 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406024 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:50:46.417586 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406027 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:50:46.417586 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406029 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:50:46.417586 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406032 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:50:46.417586 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406034 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:50:46.417586 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406037 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:50:46.417586 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406039 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:50:46.417586 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406042 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:50:46.417586 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406044 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:50:46.417586 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406047 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:50:46.417586 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406050 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:50:46.417586 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406054 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:50:46.417586 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406056 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:50:46.417586 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406059 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:50:46.417586 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406062 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:50:46.417586 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406064 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:50:46.418056 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406066 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:50:46.418056 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406069 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:50:46.418056 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406071 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:50:46.418056 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406074 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:50:46.418056 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406076 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:50:46.418056 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406079 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:50:46.418056 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406081 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:50:46.418056 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406084 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:50:46.418056 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406086 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:50:46.418056 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406089 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:50:46.418056 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406091 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:50:46.418056 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406093 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:50:46.418056 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406096 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:50:46.418056 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406100 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:50:46.418056 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406104 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:50:46.418056 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406107 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:50:46.418056 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406110 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:50:46.418056 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406112 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:50:46.418056 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406115 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:50:46.418056 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406117 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:50:46.418540 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406119 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:50:46.418540 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406122 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:50:46.418540 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406125 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:50:46.418540 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406127 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:50:46.418540 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406129 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:50:46.418540 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406132 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:50:46.418540 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406134 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:50:46.418540 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406139 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:50:46.418540 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406142 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:50:46.418540 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406146 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:50:46.418540 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406149 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:50:46.418540 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406153 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:50:46.418540 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406155 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:50:46.418540 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406158 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:50:46.418540 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406161 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:50:46.418540 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406163 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:50:46.418540 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406166 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:50:46.418540 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406169 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:50:46.418540 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406171 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:50:46.418991 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406174 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:50:46.418991 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406176 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:50:46.418991 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406179 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:50:46.418991 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406181 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:50:46.418991 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406185 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:50:46.418991 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406201 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:50:46.418991 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406204 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:50:46.418991 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406206 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:50:46.418991 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406209 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:50:46.418991 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406211 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:50:46.418991 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406214 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:50:46.418991 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406217 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:50:46.418991 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406219 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:50:46.418991 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406221 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:50:46.418991 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:46.406224 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:50:46.418991 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.406276 2577 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 01:50:46.418991 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.408240 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 01:50:46.418991 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.409329 2577 server.go:1014] "Starting client certificate rotation" Nov 03 01:50:46.418991 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.409660 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 01:50:46.419497 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.409711 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 01:50:46.419497 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.415423 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 01:50:46.419497 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.416522 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 01:50:46.425737 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.425720 2577 log.go:25] "Validated CRI v1 runtime API" Nov 03 01:50:46.428993 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.428977 2577 log.go:25] "Validated CRI v1 image API" Nov 03 01:50:46.429872 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.429849 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 01:50:46.432380 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.432362 2577 fs.go:135] Filesystem UUIDs: map[02f3caf4-3feb-44bf-8ff2-7cdce39e5298:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f55849e7-412f-4376-9f52-1bfb7706adcb:/dev/nvme0n1p3] Nov 03 01:50:46.432432 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.432380 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 01:50:46.438891 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.438784 2577 manager.go:217] Machine: {Timestamp:2026-11-03 01:50:46.437461181 +0000 UTC m=+0.325925233 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091132 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29146fb66cdab1687ed1462a4fdb15 SystemUUID:ec29146f-b66c-dab1-687e-d1462a4fdb15 BootID:6a290036-ec23-486b-b277-45db85a32cfe Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:f3:8c:47:75 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:f3:8c:47:75 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:a1:31:02:0c:cb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 01:50:46.439633 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.439622 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 01:50:46.439715 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.439699 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 01:50:46.442483 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.442457 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 01:50:46.442611 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.442484 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-69.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 01:50:46.443172 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.443162 2577 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 01:50:46.443220 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.443174 2577 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 01:50:46.443220 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.443216 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 01:50:46.443275 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.443230 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 01:50:46.444042 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.444033 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 01:50:46.446612 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.446600 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 01:50:46.446739 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.446730 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 01:50:46.447527 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.447518 2577 kubelet.go:493] "Attempting to sync node with API server" Nov 03 01:50:46.447560 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.447531 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 01:50:46.447560 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.447541 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 01:50:46.447560 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.447550 2577 kubelet.go:405] "Adding apiserver pod source" Nov 03 01:50:46.447560 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.447558 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 01:50:46.447944 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.447927 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 01:50:46.450576 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.450555 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 01:50:46.455622 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.455601 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 01:50:46.456507 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.456485 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 01:50:46.456593 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.456535 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 01:50:46.456732 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.456718 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 01:50:46.456776 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.456764 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 01:50:46.456907 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.456898 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 01:50:46.456944 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.456909 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 01:50:46.457075 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.457044 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 01:50:46.457109 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.457102 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 01:50:46.457315 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.457306 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 01:50:46.457385 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.457316 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 01:50:46.457385 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.457325 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 01:50:46.457385 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.457331 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 01:50:46.457385 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.457339 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 01:50:46.457385 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.457350 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 01:50:46.458086 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.458071 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 01:50:46.458361 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.458347 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 01:50:46.462012 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.461999 2577 server.go:1263] "Started kubelet" Nov 03 01:50:46.462123 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.462100 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 01:50:46.462177 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.462095 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 01:50:46.462177 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.462162 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 01:50:46.462746 ip-10-0-137-69 systemd[1]: Started Kubernetes Kubelet. Nov 03 01:50:46.463326 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.463305 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 01:50:46.465242 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.464498 2577 server.go:317] "Adding debug handlers to kubelet server" Nov 03 01:50:46.468091 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:46.468072 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 01:50:46.469141 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.469124 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 01:50:46.469551 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.469529 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 01:50:46.469727 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:46.469683 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-69.us-west-2.compute.internal\" not found" Nov 03 01:50:46.470225 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.470208 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 01:50:46.470225 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.470220 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 01:50:46.470225 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.470227 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 01:50:46.470415 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.470315 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 01:50:46.470415 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.470329 2577 factory.go:55] Registering systemd factory Nov 03 01:50:46.470415 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.470339 2577 factory.go:223] Registration of the systemd container factory successfully Nov 03 01:50:46.470415 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.470340 2577 reconstruct.go:97] "Volume reconstruction finished" Nov 03 01:50:46.470415 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.470350 2577 reconciler.go:29] "Reconciler: start to sync state" Nov 03 01:50:46.470634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.470594 2577 factory.go:153] Registering CRI-O factory Nov 03 01:50:46.470634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.470605 2577 factory.go:223] Registration of the crio container factory successfully Nov 03 01:50:46.470634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.470619 2577 factory.go:103] Registering Raw factory Nov 03 01:50:46.470634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.470628 2577 manager.go:1199] Started watching for new ooms in manager Nov 03 01:50:46.471779 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.471764 2577 manager.go:319] Starting recovery of all containers Nov 03 01:50:46.481028 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.481011 2577 manager.go:324] Recovery completed Nov 03 01:50:46.485476 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.485460 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:50:46.488573 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.487558 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:50:46.488573 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.487715 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:50:46.488573 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.487730 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:50:46.488733 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.488693 2577 cpu_manager.go:226] "Starting" policy="none" Nov 03 01:50:46.488733 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.488703 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 01:50:46.488733 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.488724 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 01:50:46.488945 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.488762 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:50:46.489456 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.489439 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:50:46.489539 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.489467 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:50:46.489539 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.489476 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:50:46.490147 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.490134 2577 policy_none.go:50] "Start" Nov 03 01:50:46.490211 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.490150 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 01:50:46.490211 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.490160 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 01:50:46.491224 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.491210 2577 policy_none.go:44] "Start" Nov 03 01:50:46.523697 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.523672 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 01:50:46.526039 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.526027 2577 manager.go:341] "Starting Device Plugin manager" Nov 03 01:50:46.526096 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:46.526056 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 01:50:46.526096 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.526065 2577 server.go:87] "Starting device plugin registration server" Nov 03 01:50:46.526296 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.526285 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 01:50:46.526352 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.526297 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 01:50:46.526443 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.526422 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 01:50:46.526526 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.526515 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 01:50:46.526567 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.526542 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 01:50:46.527016 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:46.526996 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 01:50:46.527086 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:46.527042 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-69.us-west-2.compute.internal\" not found" Nov 03 01:50:46.608092 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.608069 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-69.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 01:50:46.608092 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:46.608067 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-69.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 01:50:46.610926 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.610897 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 01:50:46.612037 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.612019 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 01:50:46.612037 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.612041 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 01:50:46.612185 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.612057 2577 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 01:50:46.612185 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:46.612087 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 01:50:46.626713 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.626695 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:50:46.627564 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.627548 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:50:46.627635 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.627577 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:50:46.627635 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.627588 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:50:46.627635 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.627610 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:50:46.658282 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:46.657470 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-69.us-west-2.compute.internal.18e465f31e692ecd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-69.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-69.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 01:50:46.461968077 +0000 UTC m=+0.350432129,LastTimestamp:2026-11-03 01:50:46.461968077 +0000 UTC m=+0.350432129,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-69.us-west-2.compute.internal,}" Nov 03 01:50:46.675299 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.675283 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 01:50:46.675389 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.675333 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bjr89" Nov 03 01:50:46.713057 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.713037 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal"] Nov 03 01:50:46.713133 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.713093 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:50:46.713903 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.713884 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:50:46.713979 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.713912 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:50:46.713979 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.713922 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:50:46.715105 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.715091 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:50:46.715263 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.715248 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:50:46.715326 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.715284 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:50:46.716371 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.716354 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:50:46.716464 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.716371 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:50:46.716464 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.716386 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:50:46.716464 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.716394 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:50:46.716464 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.716400 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:50:46.716464 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.716405 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:50:46.717698 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.717682 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:50:46.717789 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.717710 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:50:46.718405 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.718390 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:50:46.718489 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.718422 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:50:46.718489 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.718432 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:50:46.733551 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:46.733535 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-69.us-west-2.compute.internal\" not found" node="ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:50:46.736964 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:46.736943 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-69.us-west-2.compute.internal\" not found" node="ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:50:46.763907 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.763880 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:50:46.765525 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.765509 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:50:46.771679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.771662 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7cc04d950edb247291a58fdd1584226-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal\" (UID: \"f7cc04d950edb247291a58fdd1584226\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:50:46.771767 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.771688 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7cc04d950edb247291a58fdd1584226-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal\" (UID: \"f7cc04d950edb247291a58fdd1584226\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:50:46.771767 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.771706 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d43bad4e10a0a43dceae949566bb34d7-config\") pod \"kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal\" (UID: \"d43bad4e10a0a43dceae949566bb34d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:50:46.859395 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.859369 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:50:46.861060 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.861042 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:50:46.861341 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.861325 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:50:46.861393 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:46.861352 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-69.us-west-2.compute.internal\": node \"ip-10-0-137-69.us-west-2.compute.internal\" not found" Nov 03 01:50:46.870427 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.870413 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:50:46.872264 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.872249 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7cc04d950edb247291a58fdd1584226-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal\" (UID: \"f7cc04d950edb247291a58fdd1584226\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:50:46.872311 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.872274 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7cc04d950edb247291a58fdd1584226-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal\" (UID: \"f7cc04d950edb247291a58fdd1584226\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:50:46.872311 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.872291 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d43bad4e10a0a43dceae949566bb34d7-config\") pod \"kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal\" (UID: \"d43bad4e10a0a43dceae949566bb34d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:50:46.872383 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.872319 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7cc04d950edb247291a58fdd1584226-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal\" (UID: \"f7cc04d950edb247291a58fdd1584226\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:50:46.872383 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.872345 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d43bad4e10a0a43dceae949566bb34d7-config\") pod \"kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal\" (UID: \"d43bad4e10a0a43dceae949566bb34d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:50:46.872383 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.872346 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7cc04d950edb247291a58fdd1584226-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal\" (UID: \"f7cc04d950edb247291a58fdd1584226\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:50:46.928294 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.928242 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:50:46.928371 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.928323 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:50:46.982161 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:46.982139 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:50:47.036425 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.036403 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:50:47.040900 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.040406 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:50:47.408799 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.408733 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 01:50:47.409511 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.408893 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 01:50:47.409511 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.408921 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 01:50:47.409511 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:47.408861 2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ab176cf1b8f3e4b2b9a47658d610d089-75b99088ee9799d1.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-137-69.us-west-2.compute.internal.18e465f31ff1c5f1\": read tcp 10.0.137.69:55620->18.118.92.231:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-137-69.us-west-2.compute.internal.18e465f31ff1c5f1 default 5201 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-69.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-69.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-69.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 01:50:46 +0000 UTC,LastTimestamp:2026-11-03 01:50:46.71389933 +0000 UTC m=+0.602363384,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-69.us-west-2.compute.internal,}" Nov 03 01:50:47.448155 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.448130 2577 apiserver.go:52] "Watching apiserver" Nov 03 01:50:47.470176 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.470158 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 01:50:47.678845 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.678768 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 01:45:46 +0000 UTC" deadline="2028-04-08 16:31:33.304940042 +0000 UTC" Nov 03 01:50:47.678845 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.678795 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12542h40m45.626148073s" Nov 03 01:50:47.714487 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.714456 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 01:50:47.714796 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.714776 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-skjrl","openshift-image-registry/node-ca-ggvl5","openshift-multus/multus-g8nt7","openshift-network-diagnostics/network-check-target-rb2h2","openshift-ovn-kubernetes/ovnkube-node-mbmwx","kube-system/konnectivity-agent-bx24c","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc","openshift-cluster-node-tuning-operator/tuned-jvq7p","openshift-multus/multus-additional-cni-plugins-c6nzj","kube-system/global-pull-secret-syncer-r2z98","kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-qmlx7","openshift-network-operator/iptables-alerter-47qlq"] Nov 03 01:50:47.716366 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.716345 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-skjrl" Nov 03 01:50:47.717845 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.717821 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ggvl5" Nov 03 01:50:47.718854 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.718836 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.719963 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.719945 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:50:47.720056 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:47.720012 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:50:47.721234 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.721216 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.722519 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.722501 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bx24c" Nov 03 01:50:47.724004 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.723987 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.725665 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.725645 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.727009 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.726996 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.728165 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.728151 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:50:47.728230 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:47.728215 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:50:47.729522 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.729502 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:50:47.729623 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:47.729564 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:50:47.730777 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.730760 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-47qlq" Nov 03 01:50:47.771075 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.771059 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 01:50:47.773869 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.773853 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 01:50:47.773955 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.773858 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 01:50:47.774092 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.774075 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 01:50:47.774155 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.774074 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 01:50:47.774155 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.774106 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 01:50:47.774155 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.774133 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 01:50:47.774155 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.774145 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 01:50:47.774346 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.774171 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bqcwb\"" Nov 03 01:50:47.774346 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.774199 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zq5dk\"" Nov 03 01:50:47.774346 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.774266 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 01:50:47.774463 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.774406 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hgghc\"" Nov 03 01:50:47.774509 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.774459 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 01:50:47.774560 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.774518 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 01:50:47.774629 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.774613 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 01:50:47.774687 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.774615 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 01:50:47.774687 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.774619 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 01:50:47.774773 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.774679 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 01:50:47.774773 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.774710 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7cgvv\"" Nov 03 01:50:47.775215 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.775199 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 01:50:47.776785 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.776510 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 01:50:47.779099 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.777809 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-multus-socket-dir-parent\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.779099 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.777855 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-run-ovn\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.779099 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.777888 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1553379d-e8f5-4668-b524-13f5b93d0811-ovnkube-config\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.779099 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.777944 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-kubernetes\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.779099 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.777973 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nnhh\" (UniqueName: \"kubernetes.io/projected/61b2893f-ece1-496b-b25d-56dc1fbb1f89-kube-api-access-9nnhh\") pod \"network-metrics-daemon-qmlx7\" (UID: \"61b2893f-ece1-496b-b25d-56dc1fbb1f89\") " pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:50:47.779099 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778017 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729-iptables-alerter-script\") pod \"iptables-alerter-47qlq\" (UID: \"ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729\") " pod="openshift-network-operator/iptables-alerter-47qlq" Nov 03 01:50:47.779099 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778046 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-multus-cni-dir\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.779099 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778075 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-slash\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.779099 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778109 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1553379d-e8f5-4668-b524-13f5b93d0811-ovnkube-script-lib\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.779099 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778134 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3799f150-114a-4ed4-a955-e121465de3ab-konnectivity-ca\") pod \"konnectivity-agent-bx24c\" (UID: \"3799f150-114a-4ed4-a955-e121465de3ab\") " pod="kube-system/konnectivity-agent-bx24c" Nov 03 01:50:47.779099 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778165 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-socket-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.779099 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778208 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-lib-modules\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.779099 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778241 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd268007-d02d-44bc-8ad8-ad15135c1ae2-kubelet-config\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:50:47.779099 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778294 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95977c23-899f-4892-8b57-c0e47ea6ec82-host\") pod \"node-ca-ggvl5\" (UID: \"95977c23-899f-4892-8b57-c0e47ea6ec82\") " pod="openshift-image-registry/node-ca-ggvl5" Nov 03 01:50:47.779099 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778333 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c546c8d0-b721-45f9-b528-2e3490e84bef-cnibin\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.779099 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778374 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.779743 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778443 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-device-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.779743 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778495 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-sys-fs\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.779743 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778525 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-multus-conf-dir\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.779743 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778554 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-etc-kubernetes\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.779743 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778682 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k759x\" (UniqueName: \"kubernetes.io/projected/307c837d-2477-4460-9606-46efe63be297-kube-api-access-k759x\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.779743 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778723 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-registration-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.779743 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778765 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c546c8d0-b721-45f9-b528-2e3490e84bef-system-cni-dir\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.779743 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778797 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89-tmp-dir\") pod \"node-resolver-skjrl\" (UID: \"be3a1b21-68d9-4a7b-b1e4-30669c2dbb89\") " pod="openshift-dns/node-resolver-skjrl" Nov 03 01:50:47.779743 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778827 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-run-k8s-cni-cncf-io\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.779743 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778852 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-run-systemd\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.779743 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778880 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-cni-bin\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.779743 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778908 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-etc-selinux\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.779743 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778946 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-run\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.779743 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.778981 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkbxw\" (UniqueName: \"kubernetes.io/projected/aafc0e0e-1526-4a94-a8c0-163e80562ea8-kube-api-access-pkbxw\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.779743 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.779011 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd268007-d02d-44bc-8ad8-ad15135c1ae2-dbus\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:50:47.779743 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.779042 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-system-cni-dir\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.780276 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.779085 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3799f150-114a-4ed4-a955-e121465de3ab-agent-certs\") pod \"konnectivity-agent-bx24c\" (UID: \"3799f150-114a-4ed4-a955-e121465de3ab\") " pod="kube-system/konnectivity-agent-bx24c" Nov 03 01:50:47.780276 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.779252 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-var-lib-kubelet\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.780276 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.779354 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs\") pod \"network-metrics-daemon-qmlx7\" (UID: \"61b2893f-ece1-496b-b25d-56dc1fbb1f89\") " pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:50:47.780276 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.779380 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-etc-openvswitch\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.780276 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.779402 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-sysctl-d\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.780276 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.779443 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-systemd\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.780276 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.779500 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c546c8d0-b721-45f9-b528-2e3490e84bef-cni-binary-copy\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.780276 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.779666 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c546c8d0-b721-45f9-b528-2e3490e84bef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.780276 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.779702 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqfmb\" (UniqueName: \"kubernetes.io/projected/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729-kube-api-access-rqfmb\") pod \"iptables-alerter-47qlq\" (UID: \"ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729\") " pod="openshift-network-operator/iptables-alerter-47qlq" Nov 03 01:50:47.780276 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.779769 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzprl\" (UniqueName: \"kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl\") pod \"network-check-target-rb2h2\" (UID: \"6ffde31b-d09a-4739-9bd4-abe007571006\") " pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:50:47.780276 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.779792 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-sysctl-conf\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.780276 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.779848 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-host\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.780276 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.779885 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c7dg\" (UniqueName: \"kubernetes.io/projected/95977c23-899f-4892-8b57-c0e47ea6ec82-kube-api-access-8c7dg\") pod \"node-ca-ggvl5\" (UID: \"95977c23-899f-4892-8b57-c0e47ea6ec82\") " pod="openshift-image-registry/node-ca-ggvl5" Nov 03 01:50:47.780276 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.779931 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729-host-slash\") pod \"iptables-alerter-47qlq\" (UID: \"ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729\") " pod="openshift-network-operator/iptables-alerter-47qlq" Nov 03 01:50:47.780276 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.779958 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-var-lib-kubelet\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.780276 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.779990 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-run-multus-certs\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.780927 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.780009 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-modprobe-d\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.780927 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.780061 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c546c8d0-b721-45f9-b528-2e3490e84bef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.780927 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.780097 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-var-lib-cni-multus\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.780927 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.780134 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-var-lib-openvswitch\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.780927 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.780243 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-node-log\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.780927 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.780395 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-log-socket\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.780927 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.780470 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.780927 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.780567 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c546c8d0-b721-45f9-b528-2e3490e84bef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.780927 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.780926 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-var-lib-cni-bin\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.781294 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.780975 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-run-openvswitch\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.781294 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781030 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-tuned\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.781294 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781063 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aafc0e0e-1526-4a94-a8c0-163e80562ea8-tmp\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.781294 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781178 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xltsq\" (UniqueName: \"kubernetes.io/projected/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89-kube-api-access-xltsq\") pod \"node-resolver-skjrl\" (UID: \"be3a1b21-68d9-4a7b-b1e4-30669c2dbb89\") " pod="openshift-dns/node-resolver-skjrl" Nov 03 01:50:47.781294 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781251 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/307c837d-2477-4460-9606-46efe63be297-cni-binary-copy\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.781476 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781297 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-systemd-units\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.781476 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781330 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-run-netns\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.781476 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781390 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-run-ovn-kubernetes\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.781590 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781472 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1553379d-e8f5-4668-b524-13f5b93d0811-ovn-node-metrics-cert\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.781590 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781501 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7fh8\" (UniqueName: \"kubernetes.io/projected/1553379d-e8f5-4668-b524-13f5b93d0811-kube-api-access-s7fh8\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.781590 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781536 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/307c837d-2477-4460-9606-46efe63be297-multus-daemon-config\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.781750 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781590 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-cni-netd\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.781750 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781617 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9clj\" (UniqueName: \"kubernetes.io/projected/0155a1d8-3fb6-48da-9319-c65514404dd1-kube-api-access-v9clj\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.781750 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781640 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-sys\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.781750 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781671 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:50:47.781750 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781702 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1553379d-e8f5-4668-b524-13f5b93d0811-env-overrides\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.781924 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781762 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95977c23-899f-4892-8b57-c0e47ea6ec82-serviceca\") pod \"node-ca-ggvl5\" (UID: \"95977c23-899f-4892-8b57-c0e47ea6ec82\") " pod="openshift-image-registry/node-ca-ggvl5" Nov 03 01:50:47.781924 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781795 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89-hosts-file\") pod \"node-resolver-skjrl\" (UID: \"be3a1b21-68d9-4a7b-b1e4-30669c2dbb89\") " pod="openshift-dns/node-resolver-skjrl" Nov 03 01:50:47.781924 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781839 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-hostroot\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.781924 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781858 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-kubelet\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.781924 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781884 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-sysconfig\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.781924 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781915 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgm6s\" (UniqueName: \"kubernetes.io/projected/c546c8d0-b721-45f9-b528-2e3490e84bef-kube-api-access-mgm6s\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.782099 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781950 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-cnibin\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.782099 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.781975 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-run-netns\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.823106 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.823075 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 01:50:47.823106 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.823102 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2mmqt\"" Nov 03 01:50:47.823230 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.823119 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 01:50:47.824054 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.824039 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 01:50:47.824141 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.824125 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pc8bh\"" Nov 03 01:50:47.875030 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.875013 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 01:50:47.875136 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.875066 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cfrvs" Nov 03 01:50:47.882243 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882213 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-var-lib-openvswitch\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.882335 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882257 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-node-log\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.882335 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882288 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-log-socket\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.882335 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882301 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-var-lib-openvswitch\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.882335 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882304 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.882526 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882358 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.882526 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882364 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c546c8d0-b721-45f9-b528-2e3490e84bef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.882526 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882360 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-node-log\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.882526 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882370 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-log-socket\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.882526 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882392 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-var-lib-cni-bin\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.882526 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882477 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-var-lib-cni-bin\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.882526 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882506 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-run-openvswitch\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.882526 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882523 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c546c8d0-b721-45f9-b528-2e3490e84bef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.882892 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882532 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-tuned\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.882892 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882576 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aafc0e0e-1526-4a94-a8c0-163e80562ea8-tmp\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.882892 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882599 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-run-openvswitch\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.882892 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882768 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xltsq\" (UniqueName: \"kubernetes.io/projected/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89-kube-api-access-xltsq\") pod \"node-resolver-skjrl\" (UID: \"be3a1b21-68d9-4a7b-b1e4-30669c2dbb89\") " pod="openshift-dns/node-resolver-skjrl" Nov 03 01:50:47.882892 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882791 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/307c837d-2477-4460-9606-46efe63be297-cni-binary-copy\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.882892 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882807 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-systemd-units\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.882892 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-run-netns\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.882892 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882876 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-run-ovn-kubernetes\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.882892 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882892 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1553379d-e8f5-4668-b524-13f5b93d0811-ovn-node-metrics-cert\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.883354 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882909 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7fh8\" (UniqueName: \"kubernetes.io/projected/1553379d-e8f5-4668-b524-13f5b93d0811-kube-api-access-s7fh8\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.883354 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882930 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/307c837d-2477-4460-9606-46efe63be297-multus-daemon-config\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.883354 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882945 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-cni-netd\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.883354 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882968 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9clj\" (UniqueName: \"kubernetes.io/projected/0155a1d8-3fb6-48da-9319-c65514404dd1-kube-api-access-v9clj\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.883354 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.882995 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-sys\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.883354 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883027 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:50:47.883354 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883043 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1553379d-e8f5-4668-b524-13f5b93d0811-env-overrides\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.883354 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95977c23-899f-4892-8b57-c0e47ea6ec82-serviceca\") pod \"node-ca-ggvl5\" (UID: \"95977c23-899f-4892-8b57-c0e47ea6ec82\") " pod="openshift-image-registry/node-ca-ggvl5" Nov 03 01:50:47.883354 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883073 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89-hosts-file\") pod \"node-resolver-skjrl\" (UID: \"be3a1b21-68d9-4a7b-b1e4-30669c2dbb89\") " pod="openshift-dns/node-resolver-skjrl" Nov 03 01:50:47.883354 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883097 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-hostroot\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.883354 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883112 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-kubelet\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.883354 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883138 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-sysconfig\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.883354 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883157 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgm6s\" (UniqueName: \"kubernetes.io/projected/c546c8d0-b721-45f9-b528-2e3490e84bef-kube-api-access-mgm6s\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.883354 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883184 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-cnibin\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.883354 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883210 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-run-netns\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.883354 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883241 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-multus-socket-dir-parent\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.883354 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883264 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-run-ovn\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.883354 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883297 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1553379d-e8f5-4668-b524-13f5b93d0811-ovnkube-config\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.886634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883335 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-kubernetes\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.886634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883362 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nnhh\" (UniqueName: \"kubernetes.io/projected/61b2893f-ece1-496b-b25d-56dc1fbb1f89-kube-api-access-9nnhh\") pod \"network-metrics-daemon-qmlx7\" (UID: \"61b2893f-ece1-496b-b25d-56dc1fbb1f89\") " pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:50:47.886634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883378 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729-iptables-alerter-script\") pod \"iptables-alerter-47qlq\" (UID: \"ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729\") " pod="openshift-network-operator/iptables-alerter-47qlq" Nov 03 01:50:47.886634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883392 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-multus-cni-dir\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.886634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-slash\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.886634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883448 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1553379d-e8f5-4668-b524-13f5b93d0811-ovnkube-script-lib\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.886634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3799f150-114a-4ed4-a955-e121465de3ab-konnectivity-ca\") pod \"konnectivity-agent-bx24c\" (UID: \"3799f150-114a-4ed4-a955-e121465de3ab\") " pod="kube-system/konnectivity-agent-bx24c" Nov 03 01:50:47.886634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883486 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-socket-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.886634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-lib-modules\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.886634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883530 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd268007-d02d-44bc-8ad8-ad15135c1ae2-kubelet-config\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:50:47.886634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883546 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95977c23-899f-4892-8b57-c0e47ea6ec82-host\") pod \"node-ca-ggvl5\" (UID: \"95977c23-899f-4892-8b57-c0e47ea6ec82\") " pod="openshift-image-registry/node-ca-ggvl5" Nov 03 01:50:47.886634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883561 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c546c8d0-b721-45f9-b528-2e3490e84bef-cnibin\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.886634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883578 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.886634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-device-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.886634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-sys-fs\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.886634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883683 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-multus-conf-dir\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.886634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883699 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-etc-kubernetes\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.887484 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k759x\" (UniqueName: \"kubernetes.io/projected/307c837d-2477-4460-9606-46efe63be297-kube-api-access-k759x\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.887484 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883743 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-registration-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.887484 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c546c8d0-b721-45f9-b528-2e3490e84bef-system-cni-dir\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.887484 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883781 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89-tmp-dir\") pod \"node-resolver-skjrl\" (UID: \"be3a1b21-68d9-4a7b-b1e4-30669c2dbb89\") " pod="openshift-dns/node-resolver-skjrl" Nov 03 01:50:47.887484 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883812 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-run-k8s-cni-cncf-io\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.887484 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883848 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-run-systemd\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.887484 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883849 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/307c837d-2477-4460-9606-46efe63be297-cni-binary-copy\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.887484 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883865 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-cni-bin\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.887484 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883889 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-etc-selinux\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.887484 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883904 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-run\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.887484 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkbxw\" (UniqueName: \"kubernetes.io/projected/aafc0e0e-1526-4a94-a8c0-163e80562ea8-kube-api-access-pkbxw\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.887484 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883931 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-systemd-units\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.887484 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883940 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd268007-d02d-44bc-8ad8-ad15135c1ae2-dbus\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:50:47.887484 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.883978 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-system-cni-dir\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.887484 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884010 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3799f150-114a-4ed4-a955-e121465de3ab-agent-certs\") pod \"konnectivity-agent-bx24c\" (UID: \"3799f150-114a-4ed4-a955-e121465de3ab\") " pod="kube-system/konnectivity-agent-bx24c" Nov 03 01:50:47.887484 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884039 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-var-lib-kubelet\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.887484 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884071 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs\") pod \"network-metrics-daemon-qmlx7\" (UID: \"61b2893f-ece1-496b-b25d-56dc1fbb1f89\") " pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:50:47.888599 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884079 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-run-netns\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.888599 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884103 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-etc-openvswitch\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.888599 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884071 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-run-ovn-kubernetes\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.888599 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884131 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-sysctl-d\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.888599 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884158 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd268007-d02d-44bc-8ad8-ad15135c1ae2-dbus\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:50:47.888599 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884163 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-cni-netd\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.888599 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884128 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-system-cni-dir\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.888599 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884162 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-systemd\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.888599 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884245 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-systemd\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.888599 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884247 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c546c8d0-b721-45f9-b528-2e3490e84bef-cni-binary-copy\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.888599 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884322 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c546c8d0-b721-45f9-b528-2e3490e84bef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.888599 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884332 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 01:50:47.888599 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqfmb\" (UniqueName: \"kubernetes.io/projected/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729-kube-api-access-rqfmb\") pod \"iptables-alerter-47qlq\" (UID: \"ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729\") " pod="openshift-network-operator/iptables-alerter-47qlq" Nov 03 01:50:47.888599 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884424 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzprl\" (UniqueName: \"kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl\") pod \"network-check-target-rb2h2\" (UID: \"6ffde31b-d09a-4739-9bd4-abe007571006\") " pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:50:47.888599 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884473 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-sysctl-conf\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.888599 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884502 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-socket-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.888599 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884523 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-host\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.888599 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884647 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c7dg\" (UniqueName: \"kubernetes.io/projected/95977c23-899f-4892-8b57-c0e47ea6ec82-kube-api-access-8c7dg\") pod \"node-ca-ggvl5\" (UID: \"95977c23-899f-4892-8b57-c0e47ea6ec82\") " pod="openshift-image-registry/node-ca-ggvl5" Nov 03 01:50:47.889108 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884686 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729-host-slash\") pod \"iptables-alerter-47qlq\" (UID: \"ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729\") " pod="openshift-network-operator/iptables-alerter-47qlq" Nov 03 01:50:47.889108 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-var-lib-kubelet\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.889108 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884768 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-run-multus-certs\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.889108 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884770 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd268007-d02d-44bc-8ad8-ad15135c1ae2-kubelet-config\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:50:47.889108 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:47.884780 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:50:47.889108 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884810 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-modprobe-d\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.889108 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884815 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-var-lib-kubelet\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.889108 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c546c8d0-b721-45f9-b528-2e3490e84bef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.889108 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:47.884856 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs podName:61b2893f-ece1-496b-b25d-56dc1fbb1f89 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:48.384835814 +0000 UTC m=+2.273299881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs") pod "network-metrics-daemon-qmlx7" (UID: "61b2893f-ece1-496b-b25d-56dc1fbb1f89") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:50:47.889108 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884883 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c546c8d0-b721-45f9-b528-2e3490e84bef-cnibin\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.889108 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884889 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-var-lib-cni-multus\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.889108 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884856 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95977c23-899f-4892-8b57-c0e47ea6ec82-host\") pod \"node-ca-ggvl5\" (UID: \"95977c23-899f-4892-8b57-c0e47ea6ec82\") " pod="openshift-image-registry/node-ca-ggvl5" Nov 03 01:50:47.889108 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884929 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-var-lib-cni-multus\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.889108 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884971 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-sys\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.889108 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.885027 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3799f150-114a-4ed4-a955-e121465de3ab-konnectivity-ca\") pod \"konnectivity-agent-bx24c\" (UID: \"3799f150-114a-4ed4-a955-e121465de3ab\") " pod="kube-system/konnectivity-agent-bx24c" Nov 03 01:50:47.889108 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.885082 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.889108 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.885113 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-run-netns\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.885121 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-etc-kubernetes\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.885149 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-multus-conf-dir\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:47.885201 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:47.885302 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret podName:cd268007-d02d-44bc-8ad8-ad15135c1ae2 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:48.385284442 +0000 UTC m=+2.273748499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret") pod "global-pull-secret-syncer-r2z98" (UID: "cd268007-d02d-44bc-8ad8-ad15135c1ae2") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.885324 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89-tmp-dir\") pod \"node-resolver-skjrl\" (UID: \"be3a1b21-68d9-4a7b-b1e4-30669c2dbb89\") " pod="openshift-dns/node-resolver-skjrl" Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.885408 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-kubernetes\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.885623 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-multus-cni-dir\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.885656 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-run-k8s-cni-cncf-io\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.885667 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-slash\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.885667 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-sysctl-conf\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.885709 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-run-systemd\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.885720 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-device-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884693 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-etc-openvswitch\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.884665 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-lib-modules\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886214 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886255 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886297 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-sys-fs\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886326 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886394 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886437 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jwmm7\"" Nov 03 01:50:47.889679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886466 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 01:50:47.890418 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886485 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 01:50:47.890418 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886563 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 01:50:47.890418 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886563 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f7c55\"" Nov 03 01:50:47.890418 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886571 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:50:47.890418 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886698 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5qdp4\"" Nov 03 01:50:47.890418 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886756 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-registration-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.890418 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886757 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-sysconfig\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.890418 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886790 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-kubelet\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.890418 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886826 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89-hosts-file\") pod \"node-resolver-skjrl\" (UID: \"be3a1b21-68d9-4a7b-b1e4-30669c2dbb89\") " pod="openshift-dns/node-resolver-skjrl" Nov 03 01:50:47.890418 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886887 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-hostroot\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.890418 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886909 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c546c8d0-b721-45f9-b528-2e3490e84bef-system-cni-dir\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.890418 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886930 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729-host-slash\") pod \"iptables-alerter-47qlq\" (UID: \"ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729\") " pod="openshift-network-operator/iptables-alerter-47qlq" Nov 03 01:50:47.890418 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886949 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-host\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.890418 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886971 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-run-multus-certs\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.890418 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.886971 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-cni-bin\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.890418 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.887014 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-cnibin\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.890418 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.887056 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-var-lib-kubelet\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.890418 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.887066 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-modprobe-d\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.890418 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.887069 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-multus-socket-dir-parent\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.890418 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.887191 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-run\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.891014 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.887296 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-etc-selinux\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.891014 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.887397 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-sysctl-d\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.891014 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.887446 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-run-ovn\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.891014 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.887612 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95977c23-899f-4892-8b57-c0e47ea6ec82-serviceca\") pod \"node-ca-ggvl5\" (UID: \"95977c23-899f-4892-8b57-c0e47ea6ec82\") " pod="openshift-image-registry/node-ca-ggvl5" Nov 03 01:50:47.891014 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.887953 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c546c8d0-b721-45f9-b528-2e3490e84bef-cni-binary-copy\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.891014 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.888194 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-tuned\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.891014 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.888333 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1553379d-e8f5-4668-b524-13f5b93d0811-ovnkube-config\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.891014 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.888565 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1553379d-e8f5-4668-b524-13f5b93d0811-ovnkube-script-lib\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.891014 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.888653 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3799f150-114a-4ed4-a955-e121465de3ab-agent-certs\") pod \"konnectivity-agent-bx24c\" (UID: \"3799f150-114a-4ed4-a955-e121465de3ab\") " pod="kube-system/konnectivity-agent-bx24c" Nov 03 01:50:47.891014 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.888777 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1553379d-e8f5-4668-b524-13f5b93d0811-ovn-node-metrics-cert\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.891014 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.889415 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/307c837d-2477-4460-9606-46efe63be297-multus-daemon-config\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.891014 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.890069 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aafc0e0e-1526-4a94-a8c0-163e80562ea8-tmp\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.891014 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.890170 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1553379d-e8f5-4668-b524-13f5b93d0811-env-overrides\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.896839 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.896816 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c546c8d0-b721-45f9-b528-2e3490e84bef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.896973 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.896954 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c546c8d0-b721-45f9-b528-2e3490e84bef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:47.897038 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.896985 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729-iptables-alerter-script\") pod \"iptables-alerter-47qlq\" (UID: \"ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729\") " pod="openshift-network-operator/iptables-alerter-47qlq" Nov 03 01:50:47.940003 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.939939 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xltsq\" (UniqueName: \"kubernetes.io/projected/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89-kube-api-access-xltsq\") pod \"node-resolver-skjrl\" (UID: \"be3a1b21-68d9-4a7b-b1e4-30669c2dbb89\") " pod="openshift-dns/node-resolver-skjrl" Nov 03 01:50:47.940596 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.940571 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7fh8\" (UniqueName: \"kubernetes.io/projected/1553379d-e8f5-4668-b524-13f5b93d0811-kube-api-access-s7fh8\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:47.941441 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:47.940982 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:50:47.941441 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:47.941010 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:50:47.941441 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:47.941063 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gzprl for pod openshift-network-diagnostics/network-check-target-rb2h2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:50:47.941441 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.941069 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9clj\" (UniqueName: \"kubernetes.io/projected/0155a1d8-3fb6-48da-9319-c65514404dd1-kube-api-access-v9clj\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:47.941441 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:47.941226 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl podName:6ffde31b-d09a-4739-9bd4-abe007571006 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:48.441190667 +0000 UTC m=+2.329654726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gzprl" (UniqueName: "kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl") pod "network-check-target-rb2h2" (UID: "6ffde31b-d09a-4739-9bd4-abe007571006") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:50:47.942419 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.942392 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c7dg\" (UniqueName: \"kubernetes.io/projected/95977c23-899f-4892-8b57-c0e47ea6ec82-kube-api-access-8c7dg\") pod \"node-ca-ggvl5\" (UID: \"95977c23-899f-4892-8b57-c0e47ea6ec82\") " pod="openshift-image-registry/node-ca-ggvl5" Nov 03 01:50:47.943475 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.943456 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqfmb\" (UniqueName: \"kubernetes.io/projected/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729-kube-api-access-rqfmb\") pod \"iptables-alerter-47qlq\" (UID: \"ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729\") " pod="openshift-network-operator/iptables-alerter-47qlq" Nov 03 01:50:47.943838 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.943812 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nnhh\" (UniqueName: \"kubernetes.io/projected/61b2893f-ece1-496b-b25d-56dc1fbb1f89-kube-api-access-9nnhh\") pod \"network-metrics-daemon-qmlx7\" (UID: \"61b2893f-ece1-496b-b25d-56dc1fbb1f89\") " pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:50:47.943932 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.943915 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k759x\" (UniqueName: \"kubernetes.io/projected/307c837d-2477-4460-9606-46efe63be297-kube-api-access-k759x\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:50:47.944036 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.944018 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkbxw\" (UniqueName: \"kubernetes.io/projected/aafc0e0e-1526-4a94-a8c0-163e80562ea8-kube-api-access-pkbxw\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:47.944729 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:47.944713 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgm6s\" (UniqueName: \"kubernetes.io/projected/c546c8d0-b721-45f9-b528-2e3490e84bef-kube-api-access-mgm6s\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:48.038698 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.038672 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-skjrl" Nov 03 01:50:48.063411 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.063389 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ggvl5" Nov 03 01:50:48.069702 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.069684 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g8nt7" Nov 03 01:50:48.095235 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.095211 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:50:48.098196 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.098177 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bx24c" Nov 03 01:50:48.117427 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.117402 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:50:48.121536 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.121518 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:50:48.167138 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.167097 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:50:48.168325 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.168304 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-47qlq" Nov 03 01:50:48.168850 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:48.168822 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7cc04d950edb247291a58fdd1584226.slice/crio-f221678002fffd8191846f3e7395f648c0f6ca625f7b8c8605a7276967d46808 WatchSource:0}: Error finding container f221678002fffd8191846f3e7395f648c0f6ca625f7b8c8605a7276967d46808: Status 404 returned error can't find the container with id f221678002fffd8191846f3e7395f648c0f6ca625f7b8c8605a7276967d46808 Nov 03 01:50:48.169682 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:48.169660 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43bad4e10a0a43dceae949566bb34d7.slice/crio-9c54b0eeddd1cf0570b29b831e27d2d29c80f00ee78c0452d12575a29903b77f WatchSource:0}: Error finding container 9c54b0eeddd1cf0570b29b831e27d2d29c80f00ee78c0452d12575a29903b77f: Status 404 returned error can't find the container with id 9c54b0eeddd1cf0570b29b831e27d2d29c80f00ee78c0452d12575a29903b77f Nov 03 01:50:48.176339 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.176324 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:50:48.177562 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:48.177540 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc546c8d0_b721_45f9_b528_2e3490e84bef.slice/crio-8e990765c3265bb643e07d980763c4be6bca8d5cd63e9a8fb727c140499c0c11 WatchSource:0}: Error finding container 8e990765c3265bb643e07d980763c4be6bca8d5cd63e9a8fb727c140499c0c11: Status 404 returned error can't find the container with id 8e990765c3265bb643e07d980763c4be6bca8d5cd63e9a8fb727c140499c0c11 Nov 03 01:50:48.179152 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:48.179130 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4b2b6f_bac7_4e47_bce0_7c7ab61c8729.slice/crio-e7d1ff2e711fb13ffbc06c310106d4b6cb873440fad8ccdeb3b7f30fdd3172df WatchSource:0}: Error finding container e7d1ff2e711fb13ffbc06c310106d4b6cb873440fad8ccdeb3b7f30fdd3172df: Status 404 returned error can't find the container with id e7d1ff2e711fb13ffbc06c310106d4b6cb873440fad8ccdeb3b7f30fdd3172df Nov 03 01:50:48.372174 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.372111 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:50:48.391504 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.391470 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs\") pod \"network-metrics-daemon-qmlx7\" (UID: \"61b2893f-ece1-496b-b25d-56dc1fbb1f89\") " pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:50:48.391622 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.391519 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:50:48.391622 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:48.391603 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:50:48.391740 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:48.391629 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:50:48.391740 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:48.391649 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret podName:cd268007-d02d-44bc-8ad8-ad15135c1ae2 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:49.391636087 +0000 UTC m=+3.280100131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret") pod "global-pull-secret-syncer-r2z98" (UID: "cd268007-d02d-44bc-8ad8-ad15135c1ae2") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:50:48.391740 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:48.391689 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs podName:61b2893f-ece1-496b-b25d-56dc1fbb1f89 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:49.391670613 +0000 UTC m=+3.280134674 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs") pod "network-metrics-daemon-qmlx7" (UID: "61b2893f-ece1-496b-b25d-56dc1fbb1f89") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:50:48.476236 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.476212 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:50:48.491845 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.491818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzprl\" (UniqueName: \"kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl\") pod \"network-check-target-rb2h2\" (UID: \"6ffde31b-d09a-4739-9bd4-abe007571006\") " pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:50:48.491969 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:48.491953 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:50:48.492031 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:48.491972 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:50:48.492031 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:48.491984 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gzprl for pod openshift-network-diagnostics/network-check-target-rb2h2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:50:48.492132 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:48.492047 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl podName:6ffde31b-d09a-4739-9bd4-abe007571006 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:49.492028644 +0000 UTC m=+3.380492704 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzprl" (UniqueName: "kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl") pod "network-check-target-rb2h2" (UID: "6ffde31b-d09a-4739-9bd4-abe007571006") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:50:48.616852 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.616806 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-47qlq" event={"ID":"ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729","Type":"ContainerStarted","Data":"e7d1ff2e711fb13ffbc06c310106d4b6cb873440fad8ccdeb3b7f30fdd3172df"} Nov 03 01:50:48.618406 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.618374 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6nzj" event={"ID":"c546c8d0-b721-45f9-b528-2e3490e84bef","Type":"ContainerStarted","Data":"8e990765c3265bb643e07d980763c4be6bca8d5cd63e9a8fb727c140499c0c11"} Nov 03 01:50:48.619544 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.619517 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" event={"ID":"f7cc04d950edb247291a58fdd1584226","Type":"ContainerStarted","Data":"f221678002fffd8191846f3e7395f648c0f6ca625f7b8c8605a7276967d46808"} Nov 03 01:50:48.620543 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.620520 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal" event={"ID":"d43bad4e10a0a43dceae949566bb34d7","Type":"ContainerStarted","Data":"9c54b0eeddd1cf0570b29b831e27d2d29c80f00ee78c0452d12575a29903b77f"} Nov 03 01:50:48.877002 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.876945 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 01:45:47 +0000 UTC" deadline="2028-08-03 14:26:03.75910972 +0000 UTC" Nov 03 01:50:48.877002 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:48.876998 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15348h35m14.882114371s" Nov 03 01:50:49.004553 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:49.004519 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307c837d_2477_4460_9606_46efe63be297.slice/crio-3717a43ccb07013a2856316f7089ecbd346040aadf62076d5f53d359c368bfa6 WatchSource:0}: Error finding container 3717a43ccb07013a2856316f7089ecbd346040aadf62076d5f53d359c368bfa6: Status 404 returned error can't find the container with id 3717a43ccb07013a2856316f7089ecbd346040aadf62076d5f53d359c368bfa6 Nov 03 01:50:49.017735 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:49.017711 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95977c23_899f_4892_8b57_c0e47ea6ec82.slice/crio-20ebc1cce59a3f906431dc85e4f26347fa509a9d7dcdb0250c6c7d3a2177825e WatchSource:0}: Error finding container 20ebc1cce59a3f906431dc85e4f26347fa509a9d7dcdb0250c6c7d3a2177825e: Status 404 returned error can't find the container with id 20ebc1cce59a3f906431dc85e4f26347fa509a9d7dcdb0250c6c7d3a2177825e Nov 03 01:50:49.035455 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:49.035427 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafc0e0e_1526_4a94_a8c0_163e80562ea8.slice/crio-148d763c34aa0a3d1936e2aa1c1a5cf1f35e5a2ea449fe8e67c1917a11074350 WatchSource:0}: Error finding container 148d763c34aa0a3d1936e2aa1c1a5cf1f35e5a2ea449fe8e67c1917a11074350: Status 404 returned error can't find the container with id 148d763c34aa0a3d1936e2aa1c1a5cf1f35e5a2ea449fe8e67c1917a11074350 Nov 03 01:50:49.064732 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:49.064708 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1553379d_e8f5_4668_b524_13f5b93d0811.slice/crio-6265a52693c0f05e96393ee46c5dbad14c43c910fc567a4db682ae4dccac0fd4 WatchSource:0}: Error finding container 6265a52693c0f05e96393ee46c5dbad14c43c910fc567a4db682ae4dccac0fd4: Status 404 returned error can't find the container with id 6265a52693c0f05e96393ee46c5dbad14c43c910fc567a4db682ae4dccac0fd4 Nov 03 01:50:49.065836 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:49.065817 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3a1b21_68d9_4a7b_b1e4_30669c2dbb89.slice/crio-ee588b93a46b68acc7542cebd9b40c450fe5db9aa2ff8eadae86fa933f64a411 WatchSource:0}: Error finding container ee588b93a46b68acc7542cebd9b40c450fe5db9aa2ff8eadae86fa933f64a411: Status 404 returned error can't find the container with id ee588b93a46b68acc7542cebd9b40c450fe5db9aa2ff8eadae86fa933f64a411 Nov 03 01:50:49.080023 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:49.080003 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0155a1d8_3fb6_48da_9319_c65514404dd1.slice/crio-5d65afecf7a43d9f3f9c2bd7ebff7975998cfb0c9e385ace14617be4c88d6d2b WatchSource:0}: Error finding container 5d65afecf7a43d9f3f9c2bd7ebff7975998cfb0c9e385ace14617be4c88d6d2b: Status 404 returned error can't find the container with id 5d65afecf7a43d9f3f9c2bd7ebff7975998cfb0c9e385ace14617be4c88d6d2b Nov 03 01:50:49.087069 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:50:49.087050 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3799f150_114a_4ed4_a955_e121465de3ab.slice/crio-bf0fe3e961f1508c22d450e96ffb038537c15dbfb0ee3bb1eaf42182f0410dbc WatchSource:0}: Error finding container bf0fe3e961f1508c22d450e96ffb038537c15dbfb0ee3bb1eaf42182f0410dbc: Status 404 returned error can't find the container with id bf0fe3e961f1508c22d450e96ffb038537c15dbfb0ee3bb1eaf42182f0410dbc Nov 03 01:50:49.399031 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:49.399000 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs\") pod \"network-metrics-daemon-qmlx7\" (UID: \"61b2893f-ece1-496b-b25d-56dc1fbb1f89\") " pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:50:49.399326 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:49.399113 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:50:49.399326 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:49.399180 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:50:49.399326 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:49.399262 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs podName:61b2893f-ece1-496b-b25d-56dc1fbb1f89 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:51.399244143 +0000 UTC m=+5.287708186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs") pod "network-metrics-daemon-qmlx7" (UID: "61b2893f-ece1-496b-b25d-56dc1fbb1f89") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:50:49.399326 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:49.399261 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:50:49.399326 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:49.399302 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret podName:cd268007-d02d-44bc-8ad8-ad15135c1ae2 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:51.399294806 +0000 UTC m=+5.287758848 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret") pod "global-pull-secret-syncer-r2z98" (UID: "cd268007-d02d-44bc-8ad8-ad15135c1ae2") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:50:49.499964 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:49.499923 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzprl\" (UniqueName: \"kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl\") pod \"network-check-target-rb2h2\" (UID: \"6ffde31b-d09a-4739-9bd4-abe007571006\") " pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:50:49.500419 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:49.500129 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:50:49.500419 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:49.500155 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:50:49.500419 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:49.500170 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gzprl for pod openshift-network-diagnostics/network-check-target-rb2h2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:50:49.500419 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:49.500252 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl podName:6ffde31b-d09a-4739-9bd4-abe007571006 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:51.500232523 +0000 UTC m=+5.388696581 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzprl" (UniqueName: "kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl") pod "network-check-target-rb2h2" (UID: "6ffde31b-d09a-4739-9bd4-abe007571006") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:50:49.612759 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:49.612456 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:50:49.612759 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:49.612495 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:50:49.612759 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:49.612579 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:50:49.612759 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:49.612647 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:50:49.612759 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:49.612640 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:50:49.612759 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:49.612729 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:50:49.625222 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:49.625182 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" event={"ID":"aafc0e0e-1526-4a94-a8c0-163e80562ea8","Type":"ContainerStarted","Data":"148d763c34aa0a3d1936e2aa1c1a5cf1f35e5a2ea449fe8e67c1917a11074350"} Nov 03 01:50:49.627086 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:49.627058 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ggvl5" event={"ID":"95977c23-899f-4892-8b57-c0e47ea6ec82","Type":"ContainerStarted","Data":"20ebc1cce59a3f906431dc85e4f26347fa509a9d7dcdb0250c6c7d3a2177825e"} Nov 03 01:50:49.629129 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:49.629059 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g8nt7" event={"ID":"307c837d-2477-4460-9606-46efe63be297","Type":"ContainerStarted","Data":"3717a43ccb07013a2856316f7089ecbd346040aadf62076d5f53d359c368bfa6"} Nov 03 01:50:49.632349 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:49.632320 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-skjrl" event={"ID":"be3a1b21-68d9-4a7b-b1e4-30669c2dbb89","Type":"ContainerStarted","Data":"ee588b93a46b68acc7542cebd9b40c450fe5db9aa2ff8eadae86fa933f64a411"} Nov 03 01:50:49.634600 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:49.634493 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" event={"ID":"1553379d-e8f5-4668-b524-13f5b93d0811","Type":"ContainerStarted","Data":"6265a52693c0f05e96393ee46c5dbad14c43c910fc567a4db682ae4dccac0fd4"} Nov 03 01:50:49.635984 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:49.635904 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bx24c" event={"ID":"3799f150-114a-4ed4-a955-e121465de3ab","Type":"ContainerStarted","Data":"bf0fe3e961f1508c22d450e96ffb038537c15dbfb0ee3bb1eaf42182f0410dbc"} Nov 03 01:50:49.639313 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:49.639200 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" event={"ID":"0155a1d8-3fb6-48da-9319-c65514404dd1","Type":"ContainerStarted","Data":"5d65afecf7a43d9f3f9c2bd7ebff7975998cfb0c9e385ace14617be4c88d6d2b"} Nov 03 01:50:49.878352 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:49.878213 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 01:45:47 +0000 UTC" deadline="2028-05-30 10:28:56.104493278 +0000 UTC" Nov 03 01:50:49.878352 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:49.878301 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13784h38m6.22619726s" Nov 03 01:50:51.423103 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:51.423025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs\") pod \"network-metrics-daemon-qmlx7\" (UID: \"61b2893f-ece1-496b-b25d-56dc1fbb1f89\") " pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:50:51.423592 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:51.423107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:50:51.423592 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:51.423260 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:50:51.423592 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:51.423317 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret podName:cd268007-d02d-44bc-8ad8-ad15135c1ae2 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:55.423300333 +0000 UTC m=+9.311764378 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret") pod "global-pull-secret-syncer-r2z98" (UID: "cd268007-d02d-44bc-8ad8-ad15135c1ae2") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:50:51.423592 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:51.423379 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:50:51.423592 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:51.423410 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs podName:61b2893f-ece1-496b-b25d-56dc1fbb1f89 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:55.423399108 +0000 UTC m=+9.311863153 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs") pod "network-metrics-daemon-qmlx7" (UID: "61b2893f-ece1-496b-b25d-56dc1fbb1f89") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:50:51.524129 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:51.524093 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzprl\" (UniqueName: \"kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl\") pod \"network-check-target-rb2h2\" (UID: \"6ffde31b-d09a-4739-9bd4-abe007571006\") " pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:50:51.524313 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:51.524264 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:50:51.524313 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:51.524281 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:50:51.524313 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:51.524293 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gzprl for pod openshift-network-diagnostics/network-check-target-rb2h2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:50:51.524471 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:51.524345 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl podName:6ffde31b-d09a-4739-9bd4-abe007571006 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:55.524329106 +0000 UTC m=+9.412793148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzprl" (UniqueName: "kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl") pod "network-check-target-rb2h2" (UID: "6ffde31b-d09a-4739-9bd4-abe007571006") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:50:51.612972 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:51.612929 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:50:51.612972 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:51.612970 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:50:51.613343 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:51.612929 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:50:51.613343 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:51.613055 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:50:51.613343 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:51.613133 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:50:51.613343 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:51.613230 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:50:52.665112 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:52.665075 2577 generic.go:356] "Generic (PLEG): container finished" podID="f7cc04d950edb247291a58fdd1584226" containerID="476b28bee5b67e7ab089e8d85b053ac02144b062aff9dc8845d3361aa17deb90" exitCode=0 Nov 03 01:50:52.666124 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:52.665178 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" event={"ID":"f7cc04d950edb247291a58fdd1584226","Type":"ContainerDied","Data":"476b28bee5b67e7ab089e8d85b053ac02144b062aff9dc8845d3361aa17deb90"} Nov 03 01:50:52.678230 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:52.678034 2577 generic.go:356] "Generic (PLEG): container finished" podID="c546c8d0-b721-45f9-b528-2e3490e84bef" containerID="d4e1f327fcddc36b33052bd02575e5ab3e7e1acf3c14ad2b76019013b4aadff4" exitCode=0 Nov 03 01:50:52.678230 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:52.678077 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6nzj" event={"ID":"c546c8d0-b721-45f9-b528-2e3490e84bef","Type":"ContainerDied","Data":"d4e1f327fcddc36b33052bd02575e5ab3e7e1acf3c14ad2b76019013b4aadff4"} Nov 03 01:50:53.613171 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:53.612764 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:50:53.613171 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:53.612907 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:50:53.613171 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:53.612979 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:50:53.613171 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:53.613028 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:50:53.613171 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:53.613071 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:50:53.613171 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:53.613119 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:50:53.686772 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:53.686735 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal" event={"ID":"d43bad4e10a0a43dceae949566bb34d7","Type":"ContainerStarted","Data":"41a5aa6b885db006de39631c03c729a3718649259d829f8120b830667982f472"} Nov 03 01:50:53.693927 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:53.691416 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" event={"ID":"f7cc04d950edb247291a58fdd1584226","Type":"ContainerStarted","Data":"41d5f34f0ba4f4d748c91942a600dfc381ca36c23995f63352de22e2766ab3ed"} Nov 03 01:50:53.896361 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:53.896257 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal" podStartSLOduration=6.896237205 podStartE2EDuration="6.896237205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:53.791598335 +0000 UTC m=+7.680062407" watchObservedRunningTime="2026-11-03 01:50:53.896237205 +0000 UTC m=+7.784701269" Nov 03 01:50:54.698234 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:54.698202 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bx24c" event={"ID":"3799f150-114a-4ed4-a955-e121465de3ab","Type":"ContainerStarted","Data":"c0edb4135a39e4da5660b6d08e65821505a3d4b40591c7b01f87f6e8bbc8b041"} Nov 03 01:50:54.801480 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:54.801427 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" podStartSLOduration=7.801409355 podStartE2EDuration="7.801409355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:53.89672148 +0000 UTC m=+7.785185742" watchObservedRunningTime="2026-11-03 01:50:54.801409355 +0000 UTC m=+8.689873419" Nov 03 01:50:55.029844 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:55.029656 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bx24c" Nov 03 01:50:55.030533 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:55.030255 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bx24c" Nov 03 01:50:55.134207 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:55.134140 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bx24c" podStartSLOduration=3.59418639 podStartE2EDuration="8.134126431s" totalImagesPullingTime="4.539940041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:54.802357433 +0000 UTC m=+8.690821494" watchObservedRunningTime="2026-11-03 01:50:55.134126431 +0000 UTC m=+9.022590494" Nov 03 01:50:55.457860 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:55.457823 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs\") pod \"network-metrics-daemon-qmlx7\" (UID: \"61b2893f-ece1-496b-b25d-56dc1fbb1f89\") " pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:50:55.458043 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:55.457902 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:50:55.458120 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:55.458054 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:50:55.458120 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:55.458115 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret podName:cd268007-d02d-44bc-8ad8-ad15135c1ae2 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:03.45809772 +0000 UTC m=+17.346561766 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret") pod "global-pull-secret-syncer-r2z98" (UID: "cd268007-d02d-44bc-8ad8-ad15135c1ae2") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:50:55.458254 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:55.458183 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:50:55.458254 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:55.458234 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs podName:61b2893f-ece1-496b-b25d-56dc1fbb1f89 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:03.458220396 +0000 UTC m=+17.346684443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs") pod "network-metrics-daemon-qmlx7" (UID: "61b2893f-ece1-496b-b25d-56dc1fbb1f89") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:50:55.559083 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:55.559005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzprl\" (UniqueName: \"kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl\") pod \"network-check-target-rb2h2\" (UID: \"6ffde31b-d09a-4739-9bd4-abe007571006\") " pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:50:55.559239 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:55.559219 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:50:55.559318 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:55.559246 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:50:55.559318 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:55.559259 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gzprl for pod openshift-network-diagnostics/network-check-target-rb2h2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:50:55.559421 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:55.559334 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl podName:6ffde31b-d09a-4739-9bd4-abe007571006 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:03.559314703 +0000 UTC m=+17.447778763 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzprl" (UniqueName: "kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl") pod "network-check-target-rb2h2" (UID: "6ffde31b-d09a-4739-9bd4-abe007571006") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:50:55.613143 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:55.613119 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:50:55.613309 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:55.613288 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:50:55.613383 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:55.613352 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:50:55.613450 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:55.613429 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:50:55.613504 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:55.613471 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:50:55.613559 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:55.613527 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:50:55.705462 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:55.705427 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ggvl5" event={"ID":"95977c23-899f-4892-8b57-c0e47ea6ec82","Type":"ContainerStarted","Data":"69615bbd9746c4f0394bedb55f37a2f91f7394f369559203c932437056a9837e"} Nov 03 01:50:56.711762 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:56.711725 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" event={"ID":"0155a1d8-3fb6-48da-9319-c65514404dd1","Type":"ContainerStarted","Data":"b750bba8e663a91c908af50b7a6754c33b9f8a3b325b8213124f26800c4a1e99"} Nov 03 01:50:56.712241 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:56.711826 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 01:50:57.613540 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:57.613294 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:50:57.613728 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:57.613301 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:50:57.613728 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:57.613653 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:50:57.613728 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:57.613368 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:50:57.613897 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:57.613729 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:50:57.613897 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:57.613797 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:50:57.720926 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:57.720892 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g8nt7" event={"ID":"307c837d-2477-4460-9606-46efe63be297","Type":"ContainerStarted","Data":"1d8a432a4980f45aaca8ceb2299d9694aad4f0ebcb4a4e16e88acc8810b3acbe"} Nov 03 01:50:57.755960 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:57.755930 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 01:50:57.832080 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:57.832032 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ggvl5" podStartSLOduration=4.89117713 podStartE2EDuration="10.832020064s" totalImagesPullingTime="5.940842934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:55.809826107 +0000 UTC m=+9.698290169" watchObservedRunningTime="2026-11-03 01:50:57.832020064 +0000 UTC m=+11.720484126" Nov 03 01:50:57.832162 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:57.832107 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-g8nt7" podStartSLOduration=2.729507657 podStartE2EDuration="10.832104174s" totalImagesPullingTime="8.102596517s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:57.83161419 +0000 UTC m=+11.720078264" watchObservedRunningTime="2026-11-03 01:50:57.832104174 +0000 UTC m=+11.720568237" Nov 03 01:50:58.533321 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:58.533139 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T01:50:57.755955424Z","UUID":"1d5cfd05-801a-4e78-987c-40fbf36628bc","Handler":null,"Name":"","Endpoint":""} Nov 03 01:50:58.536101 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:58.536078 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 01:50:58.536101 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:58.536105 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 01:50:58.724749 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:58.724659 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-skjrl" event={"ID":"be3a1b21-68d9-4a7b-b1e4-30669c2dbb89","Type":"ContainerStarted","Data":"9c0f18f2fee101ef00ff60b8eee398ba7ca94a3614aac6b7317bb29be4f375fa"} Nov 03 01:50:58.726994 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:58.726939 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-47qlq" event={"ID":"ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729","Type":"ContainerStarted","Data":"7c026ff039dd00db04adcee09d2226d3f32808ecd65644e0ef2b56c88a9c2727"} Nov 03 01:50:58.729073 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:58.729035 2577 generic.go:356] "Generic (PLEG): container finished" podID="c546c8d0-b721-45f9-b528-2e3490e84bef" containerID="875ab6116e69326a51a568818deb2b1f276f42d7f4883675ca27314eaeecada4" exitCode=0 Nov 03 01:50:58.729183 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:58.729111 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6nzj" event={"ID":"c546c8d0-b721-45f9-b528-2e3490e84bef","Type":"ContainerDied","Data":"875ab6116e69326a51a568818deb2b1f276f42d7f4883675ca27314eaeecada4"} Nov 03 01:50:58.731550 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:58.731488 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" event={"ID":"0155a1d8-3fb6-48da-9319-c65514404dd1","Type":"ContainerStarted","Data":"96a936e751e9cafabcbf061a24c4dcdec0352f055bc8d3da6b8a03de3ee3bf7b"} Nov 03 01:50:58.829926 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:58.829868 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-skjrl" podStartSLOduration=3.216557301 podStartE2EDuration="11.829851359s" totalImagesPullingTime="8.613294058s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:58.829132927 +0000 UTC m=+12.717596990" watchObservedRunningTime="2026-11-03 01:50:58.829851359 +0000 UTC m=+12.718315421" Nov 03 01:50:58.862186 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:58.862160 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bx24c" Nov 03 01:50:58.862343 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:58.862301 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 01:50:58.862892 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:58.862866 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bx24c" Nov 03 01:50:59.043727 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:59.043325 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-47qlq" podStartSLOduration=3.452075413 podStartE2EDuration="12.043305373s" totalImagesPullingTime="8.59122996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:59.042831907 +0000 UTC m=+12.931295969" watchObservedRunningTime="2026-11-03 01:50:59.043305373 +0000 UTC m=+12.931769436" Nov 03 01:50:59.612747 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:59.612589 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:50:59.612890 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:59.612611 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:50:59.612890 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:59.612820 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:50:59.612890 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:59.612611 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:50:59.613034 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:59.612889 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:50:59.613034 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:50:59.612980 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:50:59.737882 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:59.735465 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" event={"ID":"0155a1d8-3fb6-48da-9319-c65514404dd1","Type":"ContainerStarted","Data":"c19d6689aa07f8e0e40e859689f5b1bca53f0affd566f84a0a420f5dbfe9e3dc"} Nov 03 01:50:59.739742 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:59.739715 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" event={"ID":"aafc0e0e-1526-4a94-a8c0-163e80562ea8","Type":"ContainerStarted","Data":"ede4fa124d7b0367a2af81ffc92eb99289ba237d3e1c222fecd32bd64f833d1c"} Nov 03 01:50:59.843018 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:59.842975 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" podStartSLOduration=2.854783248 podStartE2EDuration="12.842964066s" totalImagesPullingTime="9.988180818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:59.842512288 +0000 UTC m=+13.730976349" watchObservedRunningTime="2026-11-03 01:50:59.842964066 +0000 UTC m=+13.731428128" Nov 03 01:50:59.951779 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:50:59.951720 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" podStartSLOduration=2.5090994970000002 podStartE2EDuration="12.951702011s" totalImagesPullingTime="10.442602514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:59.950615908 +0000 UTC m=+13.839079973" watchObservedRunningTime="2026-11-03 01:50:59.951702011 +0000 UTC m=+13.840166071" Nov 03 01:51:00.744200 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:00.744157 2577 generic.go:356] "Generic (PLEG): container finished" podID="c546c8d0-b721-45f9-b528-2e3490e84bef" containerID="6718ff900de311a6af79299c38f5c55dadd99060859185e1da930154253a6d1e" exitCode=0 Nov 03 01:51:00.744651 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:00.744248 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6nzj" event={"ID":"c546c8d0-b721-45f9-b528-2e3490e84bef","Type":"ContainerDied","Data":"6718ff900de311a6af79299c38f5c55dadd99060859185e1da930154253a6d1e"} Nov 03 01:51:01.613719 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:01.613273 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:01.613719 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:01.613309 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:01.613719 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:01.613273 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:01.613719 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:01.613401 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:51:01.613719 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:01.613453 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:51:01.613719 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:01.613558 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:51:02.405385 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:02.405296 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:02.611899 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:02.611865 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:02.751717 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:02.751635 2577 generic.go:356] "Generic (PLEG): container finished" podID="c546c8d0-b721-45f9-b528-2e3490e84bef" containerID="591b3be2ce73b502fb2ee127538da7250ba2cc3ca944305e3f3281667db849bc" exitCode=0 Nov 03 01:51:02.751717 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:02.751681 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6nzj" event={"ID":"c546c8d0-b721-45f9-b528-2e3490e84bef","Type":"ContainerDied","Data":"591b3be2ce73b502fb2ee127538da7250ba2cc3ca944305e3f3281667db849bc"} Nov 03 01:51:03.521740 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:03.521707 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs\") pod \"network-metrics-daemon-qmlx7\" (UID: \"61b2893f-ece1-496b-b25d-56dc1fbb1f89\") " pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:03.522201 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:03.521771 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:03.522201 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:03.521877 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:51:03.522201 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:03.521963 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs podName:61b2893f-ece1-496b-b25d-56dc1fbb1f89 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:19.5219406 +0000 UTC m=+33.410404664 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs") pod "network-metrics-daemon-qmlx7" (UID: "61b2893f-ece1-496b-b25d-56dc1fbb1f89") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:51:03.522201 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:03.521883 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:51:03.522201 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:03.522026 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret podName:cd268007-d02d-44bc-8ad8-ad15135c1ae2 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:19.52201156 +0000 UTC m=+33.410475603 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret") pod "global-pull-secret-syncer-r2z98" (UID: "cd268007-d02d-44bc-8ad8-ad15135c1ae2") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:51:03.612471 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:03.612440 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:03.612471 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:03.612462 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:03.612641 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:03.612576 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:51:03.612687 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:03.612659 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:51:03.612744 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:03.612729 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:03.612817 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:03.612801 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:51:03.622656 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:03.622631 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzprl\" (UniqueName: \"kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl\") pod \"network-check-target-rb2h2\" (UID: \"6ffde31b-d09a-4739-9bd4-abe007571006\") " pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:03.622793 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:03.622775 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:51:03.622837 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:03.622802 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:51:03.622837 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:03.622812 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gzprl for pod openshift-network-diagnostics/network-check-target-rb2h2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:51:03.622928 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:03.622857 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl podName:6ffde31b-d09a-4739-9bd4-abe007571006 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:19.622842882 +0000 UTC m=+33.511306938 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzprl" (UniqueName: "kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl") pod "network-check-target-rb2h2" (UID: "6ffde31b-d09a-4739-9bd4-abe007571006") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:51:05.613308 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:05.613022 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:05.613906 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:05.613059 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:05.613906 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:05.613404 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:51:05.613906 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:05.613065 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:05.613906 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:05.613478 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:51:05.613906 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:05.613548 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:51:05.760065 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:05.760043 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 01:51:05.760393 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:05.760375 2577 generic.go:356] "Generic (PLEG): container finished" podID="1553379d-e8f5-4668-b524-13f5b93d0811" containerID="1ede88e3c22d1fa7301d722db396f695fab38a3ea3b8537e4e129bfff3585824" exitCode=1 Nov 03 01:51:05.760463 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:05.760442 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" event={"ID":"1553379d-e8f5-4668-b524-13f5b93d0811","Type":"ContainerStarted","Data":"24f7f415b7a9dcf96539eecc3774288151e09c1fe08d61f9425a07abce95c142"} Nov 03 01:51:05.760901 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:05.760860 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" event={"ID":"1553379d-e8f5-4668-b524-13f5b93d0811","Type":"ContainerStarted","Data":"210a7ca6d9dac7ba1671495380657f40be8ae3056c11bc0a7b2be017eb2bdb0a"} Nov 03 01:51:05.761062 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:05.761026 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" event={"ID":"1553379d-e8f5-4668-b524-13f5b93d0811","Type":"ContainerStarted","Data":"a65763d2ee5efa79bd1bcab693075df724fafd6fca19e7a087efe1da96066523"} Nov 03 01:51:05.763597 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:05.761113 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" event={"ID":"1553379d-e8f5-4668-b524-13f5b93d0811","Type":"ContainerStarted","Data":"3f84c33a2e9e018c2ba2c2cdc26c37a68de24f26cf3ab0edf8a048ad6a81707b"} Nov 03 01:51:05.763597 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:05.761177 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" event={"ID":"1553379d-e8f5-4668-b524-13f5b93d0811","Type":"ContainerDied","Data":"1ede88e3c22d1fa7301d722db396f695fab38a3ea3b8537e4e129bfff3585824"} Nov 03 01:51:05.763597 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:05.761261 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" event={"ID":"1553379d-e8f5-4668-b524-13f5b93d0811","Type":"ContainerStarted","Data":"84ac2df412e0765a762b657b707531aa63af71830ac12c17b4fcb6019ff3c526"} Nov 03 01:51:05.766000 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:05.765976 2577 generic.go:356] "Generic (PLEG): container finished" podID="c546c8d0-b721-45f9-b528-2e3490e84bef" containerID="343898d6bfc82942c83a76de21107128cd398eb46da2ff864ad5623c44eade53" exitCode=0 Nov 03 01:51:05.766090 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:05.766008 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6nzj" event={"ID":"c546c8d0-b721-45f9-b528-2e3490e84bef","Type":"ContainerDied","Data":"343898d6bfc82942c83a76de21107128cd398eb46da2ff864ad5623c44eade53"} Nov 03 01:51:06.770445 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:06.770414 2577 generic.go:356] "Generic (PLEG): container finished" podID="c546c8d0-b721-45f9-b528-2e3490e84bef" containerID="42ef1a25c50a3a583fd792e14512d164fdba4ed2e347a5c53cae457147ab75d4" exitCode=0 Nov 03 01:51:06.770445 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:06.770450 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6nzj" event={"ID":"c546c8d0-b721-45f9-b528-2e3490e84bef","Type":"ContainerDied","Data":"42ef1a25c50a3a583fd792e14512d164fdba4ed2e347a5c53cae457147ab75d4"} Nov 03 01:51:07.003664 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:07.003630 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:07.612834 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:07.612808 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:07.612994 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:07.612841 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:07.612994 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:07.612899 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:51:07.612994 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:07.612940 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:07.613141 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:07.613032 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:51:07.613141 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:07.613096 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:51:07.733318 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:07.733293 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:07.775821 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:07.775801 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 01:51:07.776164 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:07.776142 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" event={"ID":"1553379d-e8f5-4668-b524-13f5b93d0811","Type":"ContainerStarted","Data":"849d26122031f13be6c56e4f624dc83b8f5db4c984555becc747968339860e1b"} Nov 03 01:51:07.779420 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:07.779390 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6nzj" event={"ID":"c546c8d0-b721-45f9-b528-2e3490e84bef","Type":"ContainerStarted","Data":"fe12360bf45d734a5afba9a2dcb7d1a85261b901c79a828fbb65dd88d0043e03"} Nov 03 01:51:07.923885 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:07.923801 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-c6nzj" podStartSLOduration=7.915663191 podStartE2EDuration="20.923789277s" totalImagesPullingTime="13.008126086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:07.923286853 +0000 UTC m=+21.811750915" watchObservedRunningTime="2026-11-03 01:51:07.923789277 +0000 UTC m=+21.812253339" Nov 03 01:51:09.612534 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:09.612498 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:09.612534 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:09.612522 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:09.612534 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:09.612510 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:09.612964 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:09.612616 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:51:09.612964 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:09.612663 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:51:09.612964 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:09.612745 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:51:10.710952 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:10.710740 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:10.788815 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:10.788791 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 01:51:10.789127 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:10.789107 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" event={"ID":"1553379d-e8f5-4668-b524-13f5b93d0811","Type":"ContainerStarted","Data":"98a492bbe4ca706a2df59956881dc526ba3013bb0aa43e0a107d21ff283ca9d5"} Nov 03 01:51:10.789456 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:10.789431 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:10.789552 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:10.789484 2577 scope.go:122] "RemoveContainer" containerID="1ede88e3c22d1fa7301d722db396f695fab38a3ea3b8537e4e129bfff3585824" Nov 03 01:51:10.789552 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:10.789500 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:10.805093 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:10.805077 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:10.877882 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:10.875296 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:11.040590 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:11.040509 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:11.202271 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:11.202240 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:11.344857 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:11.344785 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:11.612691 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:11.612618 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:11.612691 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:11.612650 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:11.612901 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:11.612723 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:51:11.612901 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:11.612777 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:11.612901 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:11.612842 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:51:11.613045 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:11.612913 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:51:11.795343 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:11.795319 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 01:51:11.795777 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:11.795638 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" event={"ID":"1553379d-e8f5-4668-b524-13f5b93d0811","Type":"ContainerStarted","Data":"a16b154b3521fab0fd25fa332b6183160c384b5436ea5d5ac1927b9d1f57280a"} Nov 03 01:51:11.795840 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:11.795825 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:11.809989 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:11.809968 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:11.940916 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:11.940828 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" podStartSLOduration=8.940663572 podStartE2EDuration="24.94081544s" totalImagesPullingTime="16.000151868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:11.910823694 +0000 UTC m=+25.799287816" watchObservedRunningTime="2026-11-03 01:51:11.94081544 +0000 UTC m=+25.829279502" Nov 03 01:51:12.372056 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:12.371983 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:12.480306 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:12.480266 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rb2h2"] Nov 03 01:51:12.480457 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:12.480380 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:12.480519 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:12.480495 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:51:12.483677 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:12.483651 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qmlx7"] Nov 03 01:51:12.483813 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:12.483758 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:12.483890 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:12.483864 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:51:12.484268 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:12.484247 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r2z98"] Nov 03 01:51:12.484365 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:12.484345 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:12.484456 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:12.484439 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:51:12.966619 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:12.966589 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:13.612228 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:13.612185 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:13.612373 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:13.612294 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:51:14.429137 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:14.429109 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:14.613147 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:14.613123 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:14.613287 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:14.613276 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:51:14.613351 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:14.613333 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:14.613482 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:14.613462 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:51:14.662099 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:14.662073 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:14.806925 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:14.806850 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:15.214697 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:15.214542 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-skjrl_be3a1b21-68d9-4a7b-b1e4-30669c2dbb89/dns-node-resolver/0.log" Nov 03 01:51:15.612582 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:15.612519 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:15.612920 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:15.612628 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:51:15.813587 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:15.813561 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ggvl5_95977c23-899f-4892-8b57-c0e47ea6ec82/node-ca/0.log" Nov 03 01:51:15.821463 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:15.821441 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:15.971123 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:15.971099 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:16.326995 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:16.326914 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:16.485381 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:16.485353 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:16.613210 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:16.613134 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:16.613579 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:16.613231 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:51:16.613579 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:16.613323 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:16.613579 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:16.613406 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:51:16.645311 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:16.645280 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:16.974979 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:16.974949 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:17.171426 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:17.171371 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:17.501811 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:17.501779 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:17.612967 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:17.612944 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:17.613115 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:17.613036 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:51:17.777968 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:17.777895 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:17.920916 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:17.920884 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:18.386762 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:18.386732 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:18.536695 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:18.536661 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:18.612739 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:18.612713 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:18.612739 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:18.612740 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:18.612909 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:18.612805 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:51:18.612977 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:18.612952 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:51:18.679875 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:18.679851 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:18.964343 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:18.964269 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:19.122792 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:19.122762 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:19.273774 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:19.273706 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:19.542568 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:19.542491 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:19.542568 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:19.542539 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs\") pod \"network-metrics-daemon-qmlx7\" (UID: \"61b2893f-ece1-496b-b25d-56dc1fbb1f89\") " pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:19.542759 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:19.542623 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:51:19.542759 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:19.542632 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:51:19.542759 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:19.542669 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs podName:61b2893f-ece1-496b-b25d-56dc1fbb1f89 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:51.542657083 +0000 UTC m=+65.431121122 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs") pod "network-metrics-daemon-qmlx7" (UID: "61b2893f-ece1-496b-b25d-56dc1fbb1f89") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:51:19.542759 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:19.542684 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret podName:cd268007-d02d-44bc-8ad8-ad15135c1ae2 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:51.542675643 +0000 UTC m=+65.431139687 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret") pod "global-pull-secret-syncer-r2z98" (UID: "cd268007-d02d-44bc-8ad8-ad15135c1ae2") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:51:19.612384 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:19.612355 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:19.612506 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:19.612452 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:51:19.643583 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:19.643561 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzprl\" (UniqueName: \"kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl\") pod \"network-check-target-rb2h2\" (UID: \"6ffde31b-d09a-4739-9bd4-abe007571006\") " pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:19.643689 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:19.643678 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:51:19.643720 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:19.643694 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:51:19.643720 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:19.643703 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gzprl for pod openshift-network-diagnostics/network-check-target-rb2h2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:51:19.643781 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:19.643744 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl podName:6ffde31b-d09a-4739-9bd4-abe007571006 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:51.64373268 +0000 UTC m=+65.532196720 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzprl" (UniqueName: "kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl") pod "network-check-target-rb2h2" (UID: "6ffde31b-d09a-4739-9bd4-abe007571006") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:51:19.779935 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:19.779906 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:20.374132 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:20.374101 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:20.569117 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:20.569084 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:20.612303 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:20.612280 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:20.612303 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:20.612305 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:20.612474 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:20.612370 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:51:20.612474 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:20.612423 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:51:20.849345 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:20.849315 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:21.022316 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:21.022289 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:21.176146 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:21.176120 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:21.612556 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:21.612465 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:21.612986 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:21.612591 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:51:21.773964 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:21.773928 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:22.374658 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:22.374585 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:22.571080 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:22.571048 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:22.613205 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:22.613169 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:22.613604 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:22.613285 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:51:22.613604 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:22.613358 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:22.613604 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:22.613445 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:51:22.972041 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:22.972012 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:23.178263 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:23.178227 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:23.612584 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:23.612547 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:23.612748 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:23.612650 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:51:23.774679 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:23.774651 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:24.374034 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:24.374005 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:24.578651 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:24.578617 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:24.612697 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:24.612681 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:24.612819 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:24.612784 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:51:24.612873 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:24.612857 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:24.612946 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:24.612932 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:51:24.972525 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:24.972493 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:25.176493 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:25.176441 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:25.612942 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:25.612905 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:25.613108 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:25.613014 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:51:25.786754 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:25.786724 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:25.928877 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:25.928846 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:26.171237 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:26.171203 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:26.371708 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:26.371628 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:26.520446 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:26.520414 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:26.613537 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:26.613509 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:26.613681 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:26.613605 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:51:26.613719 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:26.613697 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:26.613802 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:26.613786 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:51:26.671201 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:26.671161 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:26.829136 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:26.829107 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:26.979449 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:26.979359 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:27.126016 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:27.125986 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:27.264832 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:27.264761 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:27.613110 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:27.613028 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:27.613276 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:27.613151 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:51:28.537183 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:28.537130 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:28.613292 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:28.613265 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:28.613427 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:28.613405 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:51:28.613467 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:28.613444 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:28.613554 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:28.613536 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:51:28.680392 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:28.680371 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:29.133982 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:29.133950 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:29.613064 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:29.613035 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:29.613447 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:29.613123 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:51:29.656206 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:29.656170 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:29.814403 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:29.814375 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:30.612526 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:30.612499 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:30.612686 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:30.612602 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:51:30.612686 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:30.612645 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:30.612758 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:30.612707 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:51:31.612252 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:31.612212 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:31.612635 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:31.612322 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:51:32.612693 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:32.612661 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:32.613086 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:32.612805 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:51:32.613086 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:32.612854 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:32.613086 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:32.612945 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:51:33.612425 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:33.612394 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:33.612580 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:33.612483 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:51:34.358735 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:34.358702 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:34.599769 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:34.599739 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:34.612738 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:34.612682 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:34.612849 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:34.612806 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmlx7" podUID="61b2893f-ece1-496b-b25d-56dc1fbb1f89" Nov 03 01:51:34.612908 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:34.612863 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:34.612960 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:34.612941 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb2h2" podUID="6ffde31b-d09a-4739-9bd4-abe007571006" Nov 03 01:51:34.817377 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:34.817344 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:35.094569 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:35.094535 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:35.238391 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:35.238362 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:35.424267 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:35.424235 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:35.592665 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:35.592636 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:35.612577 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:35.612557 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:35.612688 ip-10-0-137-69 kubenswrapper[2577]: E1103 01:51:35.612666 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z98" podUID="cd268007-d02d-44bc-8ad8-ad15135c1ae2" Nov 03 01:51:35.743975 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:35.743946 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:35.895543 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:35.895524 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeReady" Nov 03 01:51:35.895687 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:35.895638 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 01:51:36.038661 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.038589 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mdxfm"] Nov 03 01:51:36.063369 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.063345 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8sstb"] Nov 03 01:51:36.063494 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.063451 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:36.079468 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.079445 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mdxfm"] Nov 03 01:51:36.079568 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.079482 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8sstb"] Nov 03 01:51:36.079568 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.079545 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8sstb" Nov 03 01:51:36.082866 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.082842 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nbqt6"] Nov 03 01:51:36.097700 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.097683 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:36.113216 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.113182 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 01:51:36.113216 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.113186 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 01:51:36.113336 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.113235 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 01:51:36.113336 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.113261 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 01:51:36.113402 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.113358 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jwnvd\"" Nov 03 01:51:36.139337 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.139316 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nbqt6"] Nov 03 01:51:36.164506 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.164484 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnj6b\" (UniqueName: \"kubernetes.io/projected/cf67907b-cfa3-40ea-812d-4ea034671442-kube-api-access-xnj6b\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:36.164579 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.164511 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cf67907b-cfa3-40ea-812d-4ea034671442-crio-socket\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:36.164579 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.164554 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf67907b-cfa3-40ea-812d-4ea034671442-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:36.164667 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.164626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf67907b-cfa3-40ea-812d-4ea034671442-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:36.164667 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.164648 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf67907b-cfa3-40ea-812d-4ea034671442-data-volume\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:36.212816 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.212791 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 01:51:36.212816 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.212793 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 01:51:36.212987 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.212794 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 01:51:36.212987 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.212800 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 01:51:36.212987 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.212819 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 01:51:36.212987 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.212925 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-v5ks7\"" Nov 03 01:51:36.213102 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.213001 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bsx6s\"" Nov 03 01:51:36.265804 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.265782 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqpwg\" (UniqueName: \"kubernetes.io/projected/14044133-e020-4562-84ec-9c09ce6e6e3c-kube-api-access-nqpwg\") pod \"ingress-canary-8sstb\" (UID: \"14044133-e020-4562-84ec-9c09ce6e6e3c\") " pod="openshift-ingress-canary/ingress-canary-8sstb" Nov 03 01:51:36.265905 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.265831 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf67907b-cfa3-40ea-812d-4ea034671442-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:36.265962 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.265944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf67907b-cfa3-40ea-812d-4ea034671442-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:36.266001 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.265976 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf67907b-cfa3-40ea-812d-4ea034671442-data-volume\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:36.266050 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.266006 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f15be6a7-766d-4eae-8e57-3a34850ca41b-metrics-tls\") pod \"dns-default-nbqt6\" (UID: \"f15be6a7-766d-4eae-8e57-3a34850ca41b\") " pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:36.266050 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.266021 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52pmv\" (UniqueName: \"kubernetes.io/projected/f15be6a7-766d-4eae-8e57-3a34850ca41b-kube-api-access-52pmv\") pod \"dns-default-nbqt6\" (UID: \"f15be6a7-766d-4eae-8e57-3a34850ca41b\") " pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:36.266139 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.266087 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f15be6a7-766d-4eae-8e57-3a34850ca41b-config-volume\") pod \"dns-default-nbqt6\" (UID: \"f15be6a7-766d-4eae-8e57-3a34850ca41b\") " pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:36.266139 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.266106 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnj6b\" (UniqueName: \"kubernetes.io/projected/cf67907b-cfa3-40ea-812d-4ea034671442-kube-api-access-xnj6b\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:36.266262 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.266158 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cf67907b-cfa3-40ea-812d-4ea034671442-crio-socket\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:36.266262 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.266185 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14044133-e020-4562-84ec-9c09ce6e6e3c-cert\") pod \"ingress-canary-8sstb\" (UID: \"14044133-e020-4562-84ec-9c09ce6e6e3c\") " pod="openshift-ingress-canary/ingress-canary-8sstb" Nov 03 01:51:36.266262 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.266242 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15be6a7-766d-4eae-8e57-3a34850ca41b-tmp-dir\") pod \"dns-default-nbqt6\" (UID: \"f15be6a7-766d-4eae-8e57-3a34850ca41b\") " pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:36.266409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.266327 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf67907b-cfa3-40ea-812d-4ea034671442-data-volume\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:36.266409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.266370 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cf67907b-cfa3-40ea-812d-4ea034671442-crio-socket\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:36.266657 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.266635 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf67907b-cfa3-40ea-812d-4ea034671442-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:36.269720 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.269698 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf67907b-cfa3-40ea-812d-4ea034671442-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:36.318643 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.318597 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnj6b\" (UniqueName: \"kubernetes.io/projected/cf67907b-cfa3-40ea-812d-4ea034671442-kube-api-access-xnj6b\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:36.367358 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.367333 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f15be6a7-766d-4eae-8e57-3a34850ca41b-metrics-tls\") pod \"dns-default-nbqt6\" (UID: \"f15be6a7-766d-4eae-8e57-3a34850ca41b\") " pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:36.367497 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.367369 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52pmv\" (UniqueName: \"kubernetes.io/projected/f15be6a7-766d-4eae-8e57-3a34850ca41b-kube-api-access-52pmv\") pod \"dns-default-nbqt6\" (UID: \"f15be6a7-766d-4eae-8e57-3a34850ca41b\") " pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:36.367497 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.367405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f15be6a7-766d-4eae-8e57-3a34850ca41b-config-volume\") pod \"dns-default-nbqt6\" (UID: \"f15be6a7-766d-4eae-8e57-3a34850ca41b\") " pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:36.367497 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.367434 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14044133-e020-4562-84ec-9c09ce6e6e3c-cert\") pod \"ingress-canary-8sstb\" (UID: \"14044133-e020-4562-84ec-9c09ce6e6e3c\") " pod="openshift-ingress-canary/ingress-canary-8sstb" Nov 03 01:51:36.367497 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.367471 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15be6a7-766d-4eae-8e57-3a34850ca41b-tmp-dir\") pod \"dns-default-nbqt6\" (UID: \"f15be6a7-766d-4eae-8e57-3a34850ca41b\") " pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:36.367689 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.367497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqpwg\" (UniqueName: \"kubernetes.io/projected/14044133-e020-4562-84ec-9c09ce6e6e3c-kube-api-access-nqpwg\") pod \"ingress-canary-8sstb\" (UID: \"14044133-e020-4562-84ec-9c09ce6e6e3c\") " pod="openshift-ingress-canary/ingress-canary-8sstb" Nov 03 01:51:36.368119 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.368088 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15be6a7-766d-4eae-8e57-3a34850ca41b-tmp-dir\") pod \"dns-default-nbqt6\" (UID: \"f15be6a7-766d-4eae-8e57-3a34850ca41b\") " pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:36.368322 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.368303 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f15be6a7-766d-4eae-8e57-3a34850ca41b-config-volume\") pod \"dns-default-nbqt6\" (UID: \"f15be6a7-766d-4eae-8e57-3a34850ca41b\") " pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:36.369743 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.369720 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f15be6a7-766d-4eae-8e57-3a34850ca41b-metrics-tls\") pod \"dns-default-nbqt6\" (UID: \"f15be6a7-766d-4eae-8e57-3a34850ca41b\") " pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:36.369823 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.369804 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14044133-e020-4562-84ec-9c09ce6e6e3c-cert\") pod \"ingress-canary-8sstb\" (UID: \"14044133-e020-4562-84ec-9c09ce6e6e3c\") " pod="openshift-ingress-canary/ingress-canary-8sstb" Nov 03 01:51:36.412175 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.412151 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:36.419750 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.419722 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52pmv\" (UniqueName: \"kubernetes.io/projected/f15be6a7-766d-4eae-8e57-3a34850ca41b-kube-api-access-52pmv\") pod \"dns-default-nbqt6\" (UID: \"f15be6a7-766d-4eae-8e57-3a34850ca41b\") " pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:36.419825 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.419800 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqpwg\" (UniqueName: \"kubernetes.io/projected/14044133-e020-4562-84ec-9c09ce6e6e3c-kube-api-access-nqpwg\") pod \"ingress-canary-8sstb\" (UID: \"14044133-e020-4562-84ec-9c09ce6e6e3c\") " pod="openshift-ingress-canary/ingress-canary-8sstb" Nov 03 01:51:36.442623 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.442606 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8sstb" Nov 03 01:51:36.465409 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.465388 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:36.613140 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.613077 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:36.613280 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.613260 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:36.661709 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.661674 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 01:51:36.661709 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.661677 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-btxgv\"" Nov 03 01:51:36.661709 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.661684 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 01:51:36.661949 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.661776 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 01:51:36.661949 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.661834 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wpc48\"" Nov 03 01:51:36.911794 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.911492 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nbqt6"] Nov 03 01:51:36.911794 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.911558 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8sstb"] Nov 03 01:51:36.911794 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:36.911624 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mdxfm"] Nov 03 01:51:36.913808 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:51:36.913785 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15be6a7_766d_4eae_8e57_3a34850ca41b.slice/crio-e5c8a3898daa81e3d35949131b00ac85309812bc32d5e0a6ffa725258ab4d22b WatchSource:0}: Error finding container e5c8a3898daa81e3d35949131b00ac85309812bc32d5e0a6ffa725258ab4d22b: Status 404 returned error can't find the container with id e5c8a3898daa81e3d35949131b00ac85309812bc32d5e0a6ffa725258ab4d22b Nov 03 01:51:36.914032 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:51:36.914013 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf67907b_cfa3_40ea_812d_4ea034671442.slice/crio-c265f9fcbf01ea35d2596ad8d6f2eaa46f5c09b097a945653dd36e26ad541db4 WatchSource:0}: Error finding container c265f9fcbf01ea35d2596ad8d6f2eaa46f5c09b097a945653dd36e26ad541db4: Status 404 returned error can't find the container with id c265f9fcbf01ea35d2596ad8d6f2eaa46f5c09b097a945653dd36e26ad541db4 Nov 03 01:51:36.916937 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:51:36.916914 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14044133_e020_4562_84ec_9c09ce6e6e3c.slice/crio-68fc421f756d8b4aece98a6c530ee369e80797f733640ef2c7cb13daf048f7a2 WatchSource:0}: Error finding container 68fc421f756d8b4aece98a6c530ee369e80797f733640ef2c7cb13daf048f7a2: Status 404 returned error can't find the container with id 68fc421f756d8b4aece98a6c530ee369e80797f733640ef2c7cb13daf048f7a2 Nov 03 01:51:37.612783 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:37.612753 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:37.661477 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:37.661451 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zxsvp\"" Nov 03 01:51:37.661591 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:37.661554 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 01:51:37.846436 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:37.846407 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nbqt6" event={"ID":"f15be6a7-766d-4eae-8e57-3a34850ca41b","Type":"ContainerStarted","Data":"e5c8a3898daa81e3d35949131b00ac85309812bc32d5e0a6ffa725258ab4d22b"} Nov 03 01:51:37.847474 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:37.847453 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8sstb" event={"ID":"14044133-e020-4562-84ec-9c09ce6e6e3c","Type":"ContainerStarted","Data":"68fc421f756d8b4aece98a6c530ee369e80797f733640ef2c7cb13daf048f7a2"} Nov 03 01:51:37.848808 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:37.848792 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mdxfm" event={"ID":"cf67907b-cfa3-40ea-812d-4ea034671442","Type":"ContainerStarted","Data":"8e2da8754ae5a4c6958ca7eed68cd4e3aa46d3a2f311c7e3bb54dc668b18e50a"} Nov 03 01:51:37.848925 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:37.848826 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mdxfm" event={"ID":"cf67907b-cfa3-40ea-812d-4ea034671442","Type":"ContainerStarted","Data":"c265f9fcbf01ea35d2596ad8d6f2eaa46f5c09b097a945653dd36e26ad541db4"} Nov 03 01:51:39.690550 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:39.690510 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:39.846285 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:39.846250 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:39.855599 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:39.855570 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mdxfm" event={"ID":"cf67907b-cfa3-40ea-812d-4ea034671442","Type":"ContainerStarted","Data":"694390e2cedbe358ce96e1ecf0fcb8c25eed420da67ef4419023461031734fc1"} Nov 03 01:51:39.998045 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:39.998011 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:40.141071 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:40.141027 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:40.861018 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:40.860915 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nbqt6" event={"ID":"f15be6a7-766d-4eae-8e57-3a34850ca41b","Type":"ContainerStarted","Data":"e62a700eef90c7de1364b1b2f429c28825419d6f53c80dc23ba8b48dde8c02f0"} Nov 03 01:51:40.861018 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:40.860978 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nbqt6" event={"ID":"f15be6a7-766d-4eae-8e57-3a34850ca41b","Type":"ContainerStarted","Data":"187c6c83d5cb141882009bf0d661102f6c73ef2e162816ca1b04cafa61694d7f"} Nov 03 01:51:40.861621 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:40.861104 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:40.967764 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:40.967687 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-nbqt6" podStartSLOduration=1.900067848 podStartE2EDuration="4.967666s" totalImagesPullingTime="3.067598152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:40.966480135 +0000 UTC m=+54.854944217" watchObservedRunningTime="2026-11-03 01:51:40.967666 +0000 UTC m=+54.856130081" Nov 03 01:51:41.864761 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:41.864725 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8sstb" event={"ID":"14044133-e020-4562-84ec-9c09ce6e6e3c","Type":"ContainerStarted","Data":"6bba02a9d299edc764fcac12dc97554c150c6bf9f95d162009909215f3b49df2"} Nov 03 01:51:41.866817 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:41.866789 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mdxfm" event={"ID":"cf67907b-cfa3-40ea-812d-4ea034671442","Type":"ContainerStarted","Data":"e98afab4b121445c6bf450a05b37ada2c54eecf57bde2bf1e26d63f9b16e3917"} Nov 03 01:51:41.970213 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:41.970157 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8sstb" podStartSLOduration=2.847209878 podStartE2EDuration="6.970143938s" totalImagesPullingTime="4.12293406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:41.969943151 +0000 UTC m=+55.858407231" watchObservedRunningTime="2026-11-03 01:51:41.970143938 +0000 UTC m=+55.858607997" Nov 03 01:51:42.080735 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:42.080689 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mdxfm" podStartSLOduration=3.1921492049999998 podStartE2EDuration="7.080675711s" totalImagesPullingTime="3.888526506s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:51:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:42.080634866 +0000 UTC m=+55.969098968" watchObservedRunningTime="2026-11-03 01:51:42.080675711 +0000 UTC m=+55.969139774" Nov 03 01:51:43.810148 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:43.810118 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:44.653480 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:44.653444 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:44.804845 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:44.804802 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:45.347280 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:45.347253 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:45.490626 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:45.490583 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:45.639772 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:45.639695 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:45.793683 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:45.793646 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:45.986592 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:45.986560 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:46.153465 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.153415 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:46.293913 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.293853 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb"] Nov 03 01:51:46.295871 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.295857 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:46.298398 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.298379 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:46.335319 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.335298 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edf584ed-41f3-4fff-9e58-261b51e70b32-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rdhbb\" (UID: \"edf584ed-41f3-4fff-9e58-261b51e70b32\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:46.335405 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.335331 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/edf584ed-41f3-4fff-9e58-261b51e70b32-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rdhbb\" (UID: \"edf584ed-41f3-4fff-9e58-261b51e70b32\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:46.335405 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.335352 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edf584ed-41f3-4fff-9e58-261b51e70b32-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rdhbb\" (UID: \"edf584ed-41f3-4fff-9e58-261b51e70b32\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:46.335497 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.335478 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q7cl\" (UniqueName: \"kubernetes.io/projected/edf584ed-41f3-4fff-9e58-261b51e70b32-kube-api-access-4q7cl\") pod \"openshift-state-metrics-7bd7489957-rdhbb\" (UID: \"edf584ed-41f3-4fff-9e58-261b51e70b32\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:46.391331 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.391306 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 01:51:46.391643 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.391481 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 01:51:46.391643 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.391530 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-srm4x\"" Nov 03 01:51:46.391643 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.391535 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 01:51:46.391643 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.391627 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 01:51:46.391850 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.391774 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 01:51:46.392563 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.392546 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb"] Nov 03 01:51:46.393159 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.393141 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w7h88"] Nov 03 01:51:46.395134 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.395118 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.435926 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.435901 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edf584ed-41f3-4fff-9e58-261b51e70b32-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rdhbb\" (UID: \"edf584ed-41f3-4fff-9e58-261b51e70b32\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:46.436012 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.435933 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzpf7\" (UniqueName: \"kubernetes.io/projected/a7b495c8-cb0d-4107-b99e-3f11833410d6-kube-api-access-lzpf7\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.436012 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.435960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/edf584ed-41f3-4fff-9e58-261b51e70b32-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rdhbb\" (UID: \"edf584ed-41f3-4fff-9e58-261b51e70b32\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:46.436012 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.435984 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edf584ed-41f3-4fff-9e58-261b51e70b32-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rdhbb\" (UID: \"edf584ed-41f3-4fff-9e58-261b51e70b32\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:46.436148 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.436096 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-textfile\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.436223 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.436148 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a7b495c8-cb0d-4107-b99e-3f11833410d6-root\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.436223 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.436165 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-tls\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.436330 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.436268 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-wtmp\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.436330 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.436293 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q7cl\" (UniqueName: \"kubernetes.io/projected/edf584ed-41f3-4fff-9e58-261b51e70b32-kube-api-access-4q7cl\") pod \"openshift-state-metrics-7bd7489957-rdhbb\" (UID: \"edf584ed-41f3-4fff-9e58-261b51e70b32\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:46.436436 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.436345 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7b495c8-cb0d-4107-b99e-3f11833410d6-sys\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.436436 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.436394 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7b495c8-cb0d-4107-b99e-3f11833410d6-metrics-client-ca\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.436436 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.436418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.436560 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.436463 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-accelerators-collector-config\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.436699 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.436680 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edf584ed-41f3-4fff-9e58-261b51e70b32-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rdhbb\" (UID: \"edf584ed-41f3-4fff-9e58-261b51e70b32\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:46.438346 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.438328 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/edf584ed-41f3-4fff-9e58-261b51e70b32-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rdhbb\" (UID: \"edf584ed-41f3-4fff-9e58-261b51e70b32\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:46.438609 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.438591 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edf584ed-41f3-4fff-9e58-261b51e70b32-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rdhbb\" (UID: \"edf584ed-41f3-4fff-9e58-261b51e70b32\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:46.438695 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.438680 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq"] Nov 03 01:51:46.440880 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.440867 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:46.443653 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.443633 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 01:51:46.443871 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.443856 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8rp9w\"" Nov 03 01:51:46.444073 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.444059 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 01:51:46.445085 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.445070 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 01:51:46.453817 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.453798 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:46.485904 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.485882 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq"] Nov 03 01:51:46.534259 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.534241 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 01:51:46.536877 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.536858 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzpf7\" (UniqueName: \"kubernetes.io/projected/a7b495c8-cb0d-4107-b99e-3f11833410d6-kube-api-access-lzpf7\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.536877 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.536888 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-textfile\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.537036 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.536910 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:46.537036 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.536933 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a7b495c8-cb0d-4107-b99e-3f11833410d6-root\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.537036 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.536951 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-tls\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.537186 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.537042 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7b495c8-cb0d-4107-b99e-3f11833410d6-sys\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.537186 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.537139 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a7b495c8-cb0d-4107-b99e-3f11833410d6-root\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.537311 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.537208 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7b495c8-cb0d-4107-b99e-3f11833410d6-sys\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.537311 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.537233 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-textfile\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.537441 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.537317 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.537441 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.537358 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:46.537441 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.537404 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:46.537590 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.537508 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-wtmp\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.537590 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.537554 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srgxc\" (UniqueName: \"kubernetes.io/projected/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-api-access-srgxc\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:46.537685 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.537600 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:46.537685 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.537604 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-wtmp\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.537685 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.537627 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7b495c8-cb0d-4107-b99e-3f11833410d6-metrics-client-ca\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.537823 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.537737 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-accelerators-collector-config\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.537905 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.537887 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:46.538168 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.538142 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7b495c8-cb0d-4107-b99e-3f11833410d6-metrics-client-ca\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.539686 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.539661 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 01:51:46.559351 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.559303 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 01:51:46.559351 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.559333 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 01:51:46.559476 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.559356 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zjrsl\"" Nov 03 01:51:46.584836 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.584813 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 01:51:46.585036 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.585020 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 01:51:46.585075 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.585037 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 01:51:46.585339 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.585326 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 01:51:46.588451 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.588427 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-accelerators-collector-config\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.589773 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.589757 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-tls\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.589959 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.589936 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzpf7\" (UniqueName: \"kubernetes.io/projected/a7b495c8-cb0d-4107-b99e-3f11833410d6-kube-api-access-lzpf7\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.590074 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.590058 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.590295 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.590275 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q7cl\" (UniqueName: \"kubernetes.io/projected/edf584ed-41f3-4fff-9e58-261b51e70b32-kube-api-access-4q7cl\") pod \"openshift-state-metrics-7bd7489957-rdhbb\" (UID: \"edf584ed-41f3-4fff-9e58-261b51e70b32\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:46.600891 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.600872 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:46.645994 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.645967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:46.646127 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.646015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:46.646127 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.646075 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:46.646249 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.646124 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:46.646483 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.646443 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:46.646591 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.646520 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srgxc\" (UniqueName: \"kubernetes.io/projected/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-api-access-srgxc\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:46.646836 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.646808 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:46.646836 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.646818 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:46.647051 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.647033 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:46.648532 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.648508 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:46.648865 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.648838 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:46.655361 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.655345 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-srm4x\"" Nov 03 01:51:46.657871 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.657859 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:46.699690 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.699651 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srgxc\" (UniqueName: \"kubernetes.io/projected/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-api-access-srgxc\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:46.755329 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.755305 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8rp9w\"" Nov 03 01:51:46.757396 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.757378 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:46.758449 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.758434 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:46.769872 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:51:46.769845 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b495c8_cb0d_4107_b99e_3f11833410d6.slice/crio-25703b935a0c6e44085a51f4c26568296321b98ffb7cec9806cd5922db9785d6 WatchSource:0}: Error finding container 25703b935a0c6e44085a51f4c26568296321b98ffb7cec9806cd5922db9785d6: Status 404 returned error can't find the container with id 25703b935a0c6e44085a51f4c26568296321b98ffb7cec9806cd5922db9785d6 Nov 03 01:51:46.859873 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.859809 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:46.879653 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:46.879625 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w7h88" event={"ID":"a7b495c8-cb0d-4107-b99e-3f11833410d6","Type":"ContainerStarted","Data":"25703b935a0c6e44085a51f4c26568296321b98ffb7cec9806cd5922db9785d6"} Nov 03 01:51:47.059259 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:47.059233 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb"] Nov 03 01:51:47.092228 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:47.092184 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:47.097976 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:51:47.097957 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf584ed_41f3_4fff_9e58_261b51e70b32.slice/crio-65f689eca5dcee2d78d686333cb851aaae8738109b9cfeeb80ce366bbdfe4e20 WatchSource:0}: Error finding container 65f689eca5dcee2d78d686333cb851aaae8738109b9cfeeb80ce366bbdfe4e20: Status 404 returned error can't find the container with id 65f689eca5dcee2d78d686333cb851aaae8738109b9cfeeb80ce366bbdfe4e20 Nov 03 01:51:47.161804 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:47.161781 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq"] Nov 03 01:51:47.165309 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:51:47.165281 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37dd5ea7_75ae_4414_8c9a_90fe106cfd48.slice/crio-8bcb594a4aea99826a085c1e5d37909a2d0102fd69b180356869231de34744e9 WatchSource:0}: Error finding container 8bcb594a4aea99826a085c1e5d37909a2d0102fd69b180356869231de34744e9: Status 404 returned error can't find the container with id 8bcb594a4aea99826a085c1e5d37909a2d0102fd69b180356869231de34744e9 Nov 03 01:51:47.721749 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:47.721710 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:47.884590 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:47.884554 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" event={"ID":"edf584ed-41f3-4fff-9e58-261b51e70b32","Type":"ContainerStarted","Data":"25b07f6ef994f84c3914b21bf339ffd191352de5ef009dd9161bc27ccce61ecf"} Nov 03 01:51:47.884768 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:47.884612 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" event={"ID":"edf584ed-41f3-4fff-9e58-261b51e70b32","Type":"ContainerStarted","Data":"2864741729d9a9437b4a65c0f67518c3201205da6c86a8f8683329e7eeca5782"} Nov 03 01:51:47.884768 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:47.884632 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" event={"ID":"edf584ed-41f3-4fff-9e58-261b51e70b32","Type":"ContainerStarted","Data":"65f689eca5dcee2d78d686333cb851aaae8738109b9cfeeb80ce366bbdfe4e20"} Nov 03 01:51:47.885798 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:47.885775 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" event={"ID":"37dd5ea7-75ae-4414-8c9a-90fe106cfd48","Type":"ContainerStarted","Data":"8bcb594a4aea99826a085c1e5d37909a2d0102fd69b180356869231de34744e9"} Nov 03 01:51:48.046136 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:48.046060 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:48.561442 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:48.561399 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:48.892437 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:48.892407 2577 generic.go:356] "Generic (PLEG): container finished" podID="a7b495c8-cb0d-4107-b99e-3f11833410d6" containerID="ba101f3f99d72517c6159e89c3df35b6a55bce333f0c654c18d232f6a830dc28" exitCode=0 Nov 03 01:51:48.892841 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:48.892461 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w7h88" event={"ID":"a7b495c8-cb0d-4107-b99e-3f11833410d6","Type":"ContainerDied","Data":"ba101f3f99d72517c6159e89c3df35b6a55bce333f0c654c18d232f6a830dc28"} Nov 03 01:51:48.972289 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:48.971220 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-54564794f6-z44tk"] Nov 03 01:51:48.973972 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:48.973629 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.022441 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.022415 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-d63c6q864pbr4\"" Nov 03 01:51:49.022540 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.022466 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 01:51:49.022540 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.022524 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 01:51:49.035466 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.033957 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-54564794f6-z44tk"] Nov 03 01:51:49.064455 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.064429 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.064563 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.064478 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mbln\" (UniqueName: \"kubernetes.io/projected/fad00740-ad1a-4623-ba88-2b2319c14266-kube-api-access-2mbln\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.064563 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.064496 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-tls\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.064563 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.064516 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.064563 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.064533 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fad00740-ad1a-4623-ba88-2b2319c14266-tmp\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.064563 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.064559 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.064714 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.064582 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.064714 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.064606 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fad00740-ad1a-4623-ba88-2b2319c14266-metrics-client-ca\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.064714 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.064624 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-grpc-tls\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.123542 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.123483 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 01:51:49.123542 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.123532 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 01:51:49.123725 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.123554 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 01:51:49.123725 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.123640 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gq552\"" Nov 03 01:51:49.165466 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.165436 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.165601 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.165499 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mbln\" (UniqueName: \"kubernetes.io/projected/fad00740-ad1a-4623-ba88-2b2319c14266-kube-api-access-2mbln\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.165601 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.165533 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-tls\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.165601 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.165565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.165601 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.165590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fad00740-ad1a-4623-ba88-2b2319c14266-tmp\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.165801 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.165753 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.165861 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.165802 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.165861 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.165842 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fad00740-ad1a-4623-ba88-2b2319c14266-metrics-client-ca\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.165968 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.165876 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-grpc-tls\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.166084 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.166065 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fad00740-ad1a-4623-ba88-2b2319c14266-tmp\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.166667 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.166640 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fad00740-ad1a-4623-ba88-2b2319c14266-metrics-client-ca\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.168446 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.168421 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.168539 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.168486 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.168683 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.168662 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-tls\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.169075 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.169050 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-grpc-tls\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.170125 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.170107 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.170569 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.170544 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.219545 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.219515 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mbln\" (UniqueName: \"kubernetes.io/projected/fad00740-ad1a-4623-ba88-2b2319c14266-kube-api-access-2mbln\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.293491 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.293462 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:49.705797 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.705755 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-54564794f6-z44tk"] Nov 03 01:51:49.708267 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:51:49.708233 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad00740_ad1a_4623_ba88_2b2319c14266.slice/crio-d78d72da05776d0a39a73dac13f3863f84b23c3eef6fbde0ab624f44e7d46ed0 WatchSource:0}: Error finding container d78d72da05776d0a39a73dac13f3863f84b23c3eef6fbde0ab624f44e7d46ed0: Status 404 returned error can't find the container with id d78d72da05776d0a39a73dac13f3863f84b23c3eef6fbde0ab624f44e7d46ed0 Nov 03 01:51:49.899821 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.899787 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w7h88" event={"ID":"a7b495c8-cb0d-4107-b99e-3f11833410d6","Type":"ContainerStarted","Data":"dca79c387fb0797d01b277bdd48d3abaa6a7daf260df003626b2a68798705150"} Nov 03 01:51:49.900266 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.899849 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w7h88" event={"ID":"a7b495c8-cb0d-4107-b99e-3f11833410d6","Type":"ContainerStarted","Data":"20a09a68dfb0cbea1460d466d98e80005ef0bc7fb0c0dfbde262cd11f95af299"} Nov 03 01:51:49.902135 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.902110 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" event={"ID":"37dd5ea7-75ae-4414-8c9a-90fe106cfd48","Type":"ContainerStarted","Data":"bb7c2fe8242094ada63b21baf8091a79e12a290f64aac416964bc713dda19d97"} Nov 03 01:51:49.902271 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.902169 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" event={"ID":"37dd5ea7-75ae-4414-8c9a-90fe106cfd48","Type":"ContainerStarted","Data":"6366f4266ffabc70e8c17d53e28efaf28f538dc5516a464e1943b5fec9bcebf0"} Nov 03 01:51:49.902271 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.902202 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" event={"ID":"37dd5ea7-75ae-4414-8c9a-90fe106cfd48","Type":"ContainerStarted","Data":"f30a82175e1ec1e865093ec1d3787009bafe5251f2bfe022ef59bbe32f7a5891"} Nov 03 01:51:49.903320 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.903296 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" event={"ID":"fad00740-ad1a-4623-ba88-2b2319c14266","Type":"ContainerStarted","Data":"d78d72da05776d0a39a73dac13f3863f84b23c3eef6fbde0ab624f44e7d46ed0"} Nov 03 01:51:49.905081 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:49.905062 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" event={"ID":"edf584ed-41f3-4fff-9e58-261b51e70b32","Type":"ContainerStarted","Data":"5c12d88dba7f4338e08e16fb86e4d7b4dfeb8d74edbb2933037b74bf5c48a379"} Nov 03 01:51:50.007185 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:50.007101 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w7h88" podStartSLOduration=2.02259648 podStartE2EDuration="4.007086207s" totalImagesPullingTime="1.908913568s" totalInitContainerRuntime="75.576159ms" isStatefulPod=true podCreationTimestamp="2026-11-03 01:51:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:50.006391994 +0000 UTC m=+63.894856057" watchObservedRunningTime="2026-11-03 01:51:50.007086207 +0000 UTC m=+63.895550268" Nov 03 01:51:50.113633 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:50.113588 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" podStartSLOduration=2.335407997 podStartE2EDuration="4.113572071s" totalImagesPullingTime="1.778164074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:50.113054172 +0000 UTC m=+64.001518254" watchObservedRunningTime="2026-11-03 01:51:50.113572071 +0000 UTC m=+64.002036136" Nov 03 01:51:50.223274 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:50.223222 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" podStartSLOduration=1.9591255539999999 podStartE2EDuration="4.22320741s" totalImagesPullingTime="2.264081856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:50.222842668 +0000 UTC m=+64.111306729" watchObservedRunningTime="2026-11-03 01:51:50.22320741 +0000 UTC m=+64.111671466" Nov 03 01:51:50.352821 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:50.352742 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:50.500083 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:50.500055 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:50.710651 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:50.710618 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:50.852787 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:50.852757 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:50.869166 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:50.869140 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:51.060414 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.060350 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:51.202925 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.202896 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-98ckh"] Nov 03 01:51:51.205158 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.205141 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-98ckh" Nov 03 01:51:51.224074 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.224043 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:51.253662 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.253631 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 01:51:51.253951 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.253928 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-m7jn6\"" Nov 03 01:51:51.257988 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.257968 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-98ckh"] Nov 03 01:51:51.284507 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.284482 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fcd16c62-1c32-448a-9f84-45fb913f26ba-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-98ckh\" (UID: \"fcd16c62-1c32-448a-9f84-45fb913f26ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-98ckh" Nov 03 01:51:51.284643 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.284624 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97v2w\" (UniqueName: \"kubernetes.io/projected/fcd16c62-1c32-448a-9f84-45fb913f26ba-kube-api-access-97v2w\") pod \"monitoring-plugin-77f4f76594-98ckh\" (UID: \"fcd16c62-1c32-448a-9f84-45fb913f26ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-98ckh" Nov 03 01:51:51.386120 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.386043 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97v2w\" (UniqueName: \"kubernetes.io/projected/fcd16c62-1c32-448a-9f84-45fb913f26ba-kube-api-access-97v2w\") pod \"monitoring-plugin-77f4f76594-98ckh\" (UID: \"fcd16c62-1c32-448a-9f84-45fb913f26ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-98ckh" Nov 03 01:51:51.386120 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.386098 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fcd16c62-1c32-448a-9f84-45fb913f26ba-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-98ckh\" (UID: \"fcd16c62-1c32-448a-9f84-45fb913f26ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-98ckh" Nov 03 01:51:51.388927 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.388903 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fcd16c62-1c32-448a-9f84-45fb913f26ba-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-98ckh\" (UID: \"fcd16c62-1c32-448a-9f84-45fb913f26ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-98ckh" Nov 03 01:51:51.438911 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.438879 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97v2w\" (UniqueName: \"kubernetes.io/projected/fcd16c62-1c32-448a-9f84-45fb913f26ba-kube-api-access-97v2w\") pod \"monitoring-plugin-77f4f76594-98ckh\" (UID: \"fcd16c62-1c32-448a-9f84-45fb913f26ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-98ckh" Nov 03 01:51:51.449400 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.449373 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:51.515404 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.515377 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-98ckh" Nov 03 01:51:51.587703 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.587671 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:51.587868 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.587742 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs\") pod \"network-metrics-daemon-qmlx7\" (UID: \"61b2893f-ece1-496b-b25d-56dc1fbb1f89\") " pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:51.635883 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.635857 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 01:51:51.636000 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.635880 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 01:51:51.640462 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.640417 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:51.640540 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.640504 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs\") pod \"network-metrics-daemon-qmlx7\" (UID: \"61b2893f-ece1-496b-b25d-56dc1fbb1f89\") " pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:51.688476 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.688450 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzprl\" (UniqueName: \"kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl\") pod \"network-check-target-rb2h2\" (UID: \"6ffde31b-d09a-4739-9bd4-abe007571006\") " pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:51.697227 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.697205 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-btxgv\"" Nov 03 01:51:51.700349 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.700331 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:51.736357 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.736332 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 01:51:51.779882 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.779849 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zxsvp\"" Nov 03 01:51:51.782981 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.782954 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:51.786802 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.786781 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 01:51:51.792479 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.792429 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzprl\" (UniqueName: \"kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl\") pod \"network-check-target-rb2h2\" (UID: \"6ffde31b-d09a-4739-9bd4-abe007571006\") " pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:51.889364 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.889277 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-98ckh"] Nov 03 01:51:51.893055 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:51:51.893000 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd16c62_1c32_448a_9f84_45fb913f26ba.slice/crio-a7385d5f6115564b10316b7520473db1ca20a5b25d34d24c5d7614db270a9bd8 WatchSource:0}: Error finding container a7385d5f6115564b10316b7520473db1ca20a5b25d34d24c5d7614db270a9bd8: Status 404 returned error can't find the container with id a7385d5f6115564b10316b7520473db1ca20a5b25d34d24c5d7614db270a9bd8 Nov 03 01:51:51.913037 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:51.913006 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-98ckh" event={"ID":"fcd16c62-1c32-448a-9f84-45fb913f26ba","Type":"ContainerStarted","Data":"a7385d5f6115564b10316b7520473db1ca20a5b25d34d24c5d7614db270a9bd8"} Nov 03 01:51:52.004879 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:52.004766 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wpc48\"" Nov 03 01:51:52.007756 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:52.007734 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:52.070954 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:52.070932 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qmlx7"] Nov 03 01:51:52.073968 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:51:52.073931 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b2893f_ece1_496b_b25d_56dc1fbb1f89.slice/crio-dbdf0dca6157a139cbf3d23144097c985eed5ed94f46bd069f92d5794d24af84 WatchSource:0}: Error finding container dbdf0dca6157a139cbf3d23144097c985eed5ed94f46bd069f92d5794d24af84: Status 404 returned error can't find the container with id dbdf0dca6157a139cbf3d23144097c985eed5ed94f46bd069f92d5794d24af84 Nov 03 01:51:52.144729 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:52.144703 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r2z98"] Nov 03 01:51:52.147748 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:51:52.147714 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd268007_d02d_44bc_8ad8_ad15135c1ae2.slice/crio-42222e9da675826f9421ad9fd953680996f440356720084df669922a8311d5a7 WatchSource:0}: Error finding container 42222e9da675826f9421ad9fd953680996f440356720084df669922a8311d5a7: Status 404 returned error can't find the container with id 42222e9da675826f9421ad9fd953680996f440356720084df669922a8311d5a7 Nov 03 01:51:52.370938 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:52.370909 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rb2h2"] Nov 03 01:51:52.373716 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:51:52.373691 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ffde31b_d09a_4739_9bd4_abe007571006.slice/crio-e749d1c30334c484eccbc9087758c8f4ed5ec8903fb35c78c933da96f9227ec8 WatchSource:0}: Error finding container e749d1c30334c484eccbc9087758c8f4ed5ec8903fb35c78c933da96f9227ec8: Status 404 returned error can't find the container with id e749d1c30334c484eccbc9087758c8f4ed5ec8903fb35c78c933da96f9227ec8 Nov 03 01:51:52.741905 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:52.741867 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:52.888569 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:52.888531 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:52.918349 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:52.918313 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r2z98" event={"ID":"cd268007-d02d-44bc-8ad8-ad15135c1ae2","Type":"ContainerStarted","Data":"42222e9da675826f9421ad9fd953680996f440356720084df669922a8311d5a7"} Nov 03 01:51:52.919776 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:52.919748 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rb2h2" event={"ID":"6ffde31b-d09a-4739-9bd4-abe007571006","Type":"ContainerStarted","Data":"e749d1c30334c484eccbc9087758c8f4ed5ec8903fb35c78c933da96f9227ec8"} Nov 03 01:51:52.921225 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:52.921171 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qmlx7" event={"ID":"61b2893f-ece1-496b-b25d-56dc1fbb1f89","Type":"ContainerStarted","Data":"dbdf0dca6157a139cbf3d23144097c985eed5ed94f46bd069f92d5794d24af84"} Nov 03 01:51:53.094671 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:53.094603 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:53.249544 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:53.249507 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:53.933520 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:53.933477 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" event={"ID":"fad00740-ad1a-4623-ba88-2b2319c14266","Type":"ContainerStarted","Data":"0064a7d42c0254f6489b71939e5940f00ade51177695af2ead03ddceada9f35b"} Nov 03 01:51:53.933622 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:53.933577 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" event={"ID":"fad00740-ad1a-4623-ba88-2b2319c14266","Type":"ContainerStarted","Data":"39f3c7d390c9a749b98d3cf38e0ff438f9729378bd1ff5e284f52cf6e67efb7c"} Nov 03 01:51:53.933687 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:53.933622 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" event={"ID":"fad00740-ad1a-4623-ba88-2b2319c14266","Type":"ContainerStarted","Data":"7d5dd750165ce034320eaff260d00af37a18557ba48eac3cceb258014186321c"} Nov 03 01:51:54.786031 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:54.785958 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:54.942059 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:54.941213 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qmlx7" event={"ID":"61b2893f-ece1-496b-b25d-56dc1fbb1f89","Type":"ContainerStarted","Data":"1f12d661abc848d59e0bafd557be1329336d7904b7396cb42c486a1a3861a1b5"} Nov 03 01:51:54.942059 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:54.941288 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qmlx7" event={"ID":"61b2893f-ece1-496b-b25d-56dc1fbb1f89","Type":"ContainerStarted","Data":"b896b8ac86ef92a606766f16d39291d63399d8b9e9413749fcd57165cdd3ed5a"} Nov 03 01:51:54.944892 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:54.944148 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-98ckh" event={"ID":"fcd16c62-1c32-448a-9f84-45fb913f26ba","Type":"ContainerStarted","Data":"879d9ae5af6c1a66497fccb50eece9124cceddf9d8829cdef7039955446fefd0"} Nov 03 01:51:54.944892 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:54.944827 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-98ckh" Nov 03 01:51:54.947314 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:54.947180 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:54.952564 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:54.952433 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-98ckh" Nov 03 01:51:54.956699 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:54.955730 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" event={"ID":"fad00740-ad1a-4623-ba88-2b2319c14266","Type":"ContainerStarted","Data":"d14952a4f56dfceaf529c5b5a10b70deb2e8fecbdccb5168c22a576c53435c7f"} Nov 03 01:51:54.956699 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:54.955800 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" event={"ID":"fad00740-ad1a-4623-ba88-2b2319c14266","Type":"ContainerStarted","Data":"64b32af2cbacee433acd91d716f5370d851783c4803826df64200f5e8fe9a120"} Nov 03 01:51:54.956699 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:54.955842 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" event={"ID":"fad00740-ad1a-4623-ba88-2b2319c14266","Type":"ContainerStarted","Data":"8be901fd03345c9f6eed674a0ec990387f03fb62ddb178450f663d964d234705"} Nov 03 01:51:54.956699 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:54.956622 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:55.051676 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.051617 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qmlx7" podStartSLOduration=66.275403093 podStartE2EDuration="1m8.051599783s" totalImagesPullingTime="1.77619669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:55.051078154 +0000 UTC m=+68.939542215" watchObservedRunningTime="2026-11-03 01:51:55.051599783 +0000 UTC m=+68.940063845" Nov 03 01:51:55.173812 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.173763 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" podStartSLOduration=2.367898915 podStartE2EDuration="7.1737473s" totalImagesPullingTime="4.805848385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:55.171123527 +0000 UTC m=+69.059587616" watchObservedRunningTime="2026-11-03 01:51:55.1737473 +0000 UTC m=+69.062211362" Nov 03 01:51:55.245736 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.245650 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:55.251661 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.251633 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 01:51:55.254909 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.254888 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.280894 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.280858 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-98ckh" podStartSLOduration=1.949734136 podStartE2EDuration="4.280842392s" totalImagesPullingTime="2.331108256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:55.280376874 +0000 UTC m=+69.168840963" watchObservedRunningTime="2026-11-03 01:51:55.280842392 +0000 UTC m=+69.169306456" Nov 03 01:51:55.305441 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.305392 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 01:51:55.305441 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.305417 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 01:51:55.305606 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.305486 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 01:51:55.305718 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.305698 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 01:51:55.322619 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.322590 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.322743 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.322644 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.322743 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.322695 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-config\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.322743 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.322729 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a134e24f-d1ad-467a-914e-0ff35ca0177b-config-out\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.322903 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.322754 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.322903 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.322818 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.322903 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.322870 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.322903 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.322898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.323084 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.322927 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.323084 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.322955 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.323084 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.322981 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv668\" (UniqueName: \"kubernetes.io/projected/a134e24f-d1ad-467a-914e-0ff35ca0177b-kube-api-access-sv668\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.323084 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.323075 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.323571 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.323106 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-web-config\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.323571 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.323133 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.323571 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.323163 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.323571 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.323240 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.323571 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.323265 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.323571 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.323291 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a134e24f-d1ad-467a-914e-0ff35ca0177b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.323571 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.323315 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.323571 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.323341 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.373661 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.373635 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 01:51:55.373789 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.373679 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 01:51:55.375619 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.375595 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 01:51:55.398823 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.398679 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 01:51:55.398823 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.398703 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 01:51:55.398823 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.398703 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 01:51:55.398823 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.398756 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 01:51:55.398823 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.398764 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 01:51:55.398823 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.398797 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 01:51:55.399750 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.399668 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 01:51:55.399871 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.399829 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 01:51:55.400078 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.400056 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nj9dq\"" Nov 03 01:51:55.400185 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.400153 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 01:51:55.400675 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.400657 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8rsahm0n9sh3l\"" Nov 03 01:51:55.405751 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.405724 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:55.424134 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.424115 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-config\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.424234 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.424144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a134e24f-d1ad-467a-914e-0ff35ca0177b-config-out\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.424234 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.424161 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.424234 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.424182 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.424400 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.424354 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.424400 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.424389 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.424494 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.424419 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.424494 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.424447 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.424494 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.424470 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv668\" (UniqueName: \"kubernetes.io/projected/a134e24f-d1ad-467a-914e-0ff35ca0177b-kube-api-access-sv668\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.424634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.424496 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.424634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.424518 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-web-config\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.424634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.424535 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.424634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.424551 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.424634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.424571 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.424634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.424585 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.424634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.424601 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a134e24f-d1ad-467a-914e-0ff35ca0177b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.424634 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.424615 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.424950 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.424638 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.424950 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.424701 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.424950 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.424733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.425131 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.425108 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.426774 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.425468 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.426774 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.426083 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.426774 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.426405 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.426774 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.426425 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.427458 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.427065 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a134e24f-d1ad-467a-914e-0ff35ca0177b-config-out\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.427652 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.427630 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-config\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.428292 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.428017 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.428622 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.428601 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.428760 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.428665 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.428980 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.428959 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.429733 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.429652 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.430238 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.430218 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.431098 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.431074 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.431507 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.431484 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.431626 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.431598 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-web-config\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.432061 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.432039 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.432426 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.432405 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.432704 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.432688 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a134e24f-d1ad-467a-914e-0ff35ca0177b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.479091 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.479065 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv668\" (UniqueName: \"kubernetes.io/projected/a134e24f-d1ad-467a-914e-0ff35ca0177b-kube-api-access-sv668\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:55.576313 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:55.576240 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:56.001282 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:56.001249 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 01:51:56.005736 ip-10-0-137-69 kubenswrapper[2577]: W1103 01:51:56.005702 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda134e24f_d1ad_467a_914e_0ff35ca0177b.slice/crio-950008f8e686ec9f369a11655f4dd77308b491c84e4912ddded048cc3bfbccee WatchSource:0}: Error finding container 950008f8e686ec9f369a11655f4dd77308b491c84e4912ddded048cc3bfbccee: Status 404 returned error can't find the container with id 950008f8e686ec9f369a11655f4dd77308b491c84e4912ddded048cc3bfbccee Nov 03 01:51:56.955638 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:56.955602 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:56.968536 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:56.968398 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a134e24f-d1ad-467a-914e-0ff35ca0177b","Type":"ContainerStarted","Data":"950008f8e686ec9f369a11655f4dd77308b491c84e4912ddded048cc3bfbccee"} Nov 03 01:51:57.193383 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:57.193304 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:57.603266 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:57.603233 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:57.752183 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:57.752147 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:57.950685 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:57.950650 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:57.973869 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:57.973840 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rb2h2" event={"ID":"6ffde31b-d09a-4739-9bd4-abe007571006","Type":"ContainerStarted","Data":"1ce46bf46fc63d78a15fb81449217b05ddd0b1238ae9c6407b6888175c420b26"} Nov 03 01:51:57.974020 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:57.973927 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:57.975378 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:57.975351 2577 generic.go:356] "Generic (PLEG): container finished" podID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerID="e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066" exitCode=0 Nov 03 01:51:57.975466 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:57.975424 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a134e24f-d1ad-467a-914e-0ff35ca0177b","Type":"ContainerDied","Data":"e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066"} Nov 03 01:51:58.083653 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:58.083600 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rb2h2" podStartSLOduration=66.541462026 podStartE2EDuration="1m11.083580095s" totalImagesPullingTime="4.542118069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:58.081907552 +0000 UTC m=+71.970371615" watchObservedRunningTime="2026-11-03 01:51:58.083580095 +0000 UTC m=+71.972044159" Nov 03 01:51:58.147696 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:58.147656 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:58.259452 ip-10-0-137-69 kubenswrapper[2577]: I1103 01:51:58.259294 2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 01:51:58.259466 ip-10-0-137-69 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 01:51:58.273667 ip-10-0-137-69 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 01:51:58.273840 ip-10-0-137-69 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 01:51:58.273887 ip-10-0-137-69 systemd[1]: kubelet.service: Consumed 3.922s CPU time. Nov 03 01:51:58.283789 ip-10-0-137-69 systemd[1]: Starting Kubernetes Kubelet... Nov 03 01:51:58.344864 ip-10-0-137-69 kubenswrapper[5463]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:51:58.344864 ip-10-0-137-69 kubenswrapper[5463]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 01:51:58.344864 ip-10-0-137-69 kubenswrapper[5463]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:51:58.344864 ip-10-0-137-69 kubenswrapper[5463]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:51:58.346884 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346807 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:51:58.346884 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346881 5463 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:51:58.346884 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346885 5463 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:51:58.346884 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346888 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:51:58.346884 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346892 5463 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:51:58.347070 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346895 5463 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:51:58.347070 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346898 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:51:58.347070 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346901 5463 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:51:58.347070 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346903 5463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:51:58.347070 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346906 5463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:51:58.347070 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346909 5463 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:51:58.347070 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346911 5463 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:51:58.347070 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346915 5463 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:51:58.347070 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346918 5463 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:51:58.347070 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346920 5463 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:51:58.347070 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346927 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:51:58.347070 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346930 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:51:58.347070 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346932 5463 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:51:58.347070 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346935 5463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:51:58.347070 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346938 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:51:58.347070 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346940 5463 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:51:58.347070 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346943 5463 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:51:58.347070 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346946 5463 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:51:58.347070 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346949 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:51:58.347539 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346951 5463 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:51:58.347539 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346954 5463 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:51:58.347539 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346956 5463 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:51:58.347539 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346959 5463 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:51:58.347539 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346963 5463 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:51:58.347539 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346967 5463 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:51:58.347539 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346969 5463 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:51:58.347539 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346972 5463 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:51:58.347539 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346974 5463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:51:58.347539 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346983 5463 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:51:58.347539 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346987 5463 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:51:58.347539 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346990 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:51:58.347539 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346992 5463 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:51:58.347539 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346997 5463 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:51:58.347539 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.346999 5463 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:51:58.347539 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347002 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:51:58.347539 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347005 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:51:58.347539 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347008 5463 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:51:58.347966 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347010 5463 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:51:58.347966 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347013 5463 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:51:58.347966 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347016 5463 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:51:58.347966 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347018 5463 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:51:58.347966 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347021 5463 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:51:58.347966 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347023 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:51:58.347966 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347027 5463 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:51:58.347966 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347029 5463 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:51:58.347966 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347032 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:51:58.347966 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347034 5463 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:51:58.347966 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347037 5463 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:51:58.347966 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347040 5463 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:51:58.347966 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347042 5463 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:51:58.347966 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347045 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:51:58.347966 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347047 5463 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:51:58.347966 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347050 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:51:58.347966 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347053 5463 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:51:58.347966 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347055 5463 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:51:58.347966 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347058 5463 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:51:58.347966 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347060 5463 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:51:58.348474 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347063 5463 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:51:58.348474 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347065 5463 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:51:58.348474 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347067 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:51:58.348474 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347070 5463 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:51:58.348474 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347073 5463 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:51:58.348474 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347075 5463 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:51:58.348474 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347078 5463 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:51:58.348474 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347081 5463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:51:58.348474 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347084 5463 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:51:58.348474 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347087 5463 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:51:58.348474 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347089 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:51:58.348474 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347092 5463 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:51:58.348474 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347095 5463 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:51:58.348474 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347097 5463 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:51:58.348474 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347101 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:51:58.348474 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347104 5463 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:51:58.348474 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347106 5463 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:51:58.348474 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347109 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:51:58.348474 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347111 5463 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:51:58.348474 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347115 5463 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:51:58.348949 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347117 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:51:58.348949 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347120 5463 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:51:58.348949 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347123 5463 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:51:58.348949 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347125 5463 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:51:58.348949 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347128 5463 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:51:58.348949 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347130 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:51:58.348949 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347132 5463 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:51:58.348949 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347135 5463 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:51:58.348949 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347137 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:51:58.348949 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347140 5463 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:51:58.348949 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347142 5463 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:51:58.348949 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347145 5463 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:51:58.348949 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347147 5463 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:51:58.348949 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347150 5463 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:51:58.348949 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347153 5463 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:51:58.348949 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347155 5463 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:51:58.348949 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347158 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:51:58.348949 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347160 5463 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:51:58.348949 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347163 5463 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:51:58.349419 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347167 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:51:58.349419 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347170 5463 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:51:58.349419 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347172 5463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:51:58.349419 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347175 5463 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:51:58.349419 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347177 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:51:58.349419 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347928 5463 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:51:58.349419 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347935 5463 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:51:58.349419 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347938 5463 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:51:58.349419 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347941 5463 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:51:58.349419 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347944 5463 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:51:58.349419 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347946 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:51:58.349419 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347949 5463 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:51:58.349419 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347951 5463 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:51:58.349419 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347955 5463 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:51:58.349419 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347958 5463 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:51:58.349419 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347960 5463 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:51:58.349419 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347963 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:51:58.349419 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347966 5463 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:51:58.349419 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347969 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:51:58.349896 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347971 5463 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:51:58.349896 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347974 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:51:58.349896 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347977 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:51:58.349896 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347980 5463 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:51:58.349896 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347982 5463 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:51:58.349896 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347985 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:51:58.349896 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347987 5463 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:51:58.349896 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347990 5463 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:51:58.349896 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347992 5463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:51:58.349896 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347995 5463 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:51:58.349896 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.347997 5463 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:51:58.349896 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348000 5463 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:51:58.349896 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348003 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:51:58.349896 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348006 5463 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:51:58.349896 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348009 5463 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:51:58.349896 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348012 5463 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:51:58.349896 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348015 5463 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:51:58.349896 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348017 5463 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:51:58.349896 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348020 5463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:51:58.349896 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348023 5463 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:51:58.350396 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348025 5463 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:51:58.350396 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348028 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:51:58.350396 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348031 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:51:58.350396 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348033 5463 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:51:58.350396 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348036 5463 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:51:58.350396 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348039 5463 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:51:58.350396 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348042 5463 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:51:58.350396 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348045 5463 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:51:58.350396 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348048 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:51:58.350396 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348051 5463 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:51:58.350396 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348053 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:51:58.350396 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348056 5463 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:51:58.350396 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348058 5463 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:51:58.350396 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348061 5463 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:51:58.350396 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348063 5463 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:51:58.350396 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348066 5463 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:51:58.350396 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348069 5463 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:51:58.350396 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348071 5463 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:51:58.350396 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348073 5463 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:51:58.350889 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348076 5463 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:51:58.350889 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348079 5463 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:51:58.350889 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348081 5463 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:51:58.350889 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348084 5463 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:51:58.350889 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348086 5463 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:51:58.350889 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348090 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:51:58.350889 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348092 5463 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:51:58.350889 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348095 5463 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:51:58.350889 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348100 5463 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:51:58.350889 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348103 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:51:58.350889 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348106 5463 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:51:58.350889 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348108 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:51:58.350889 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348111 5463 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:51:58.350889 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348114 5463 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:51:58.350889 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348118 5463 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:51:58.350889 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348121 5463 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:51:58.350889 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348124 5463 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:51:58.350889 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348126 5463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:51:58.350889 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348129 5463 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:51:58.351355 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348131 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:51:58.351355 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348134 5463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:51:58.351355 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348140 5463 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:51:58.351355 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348143 5463 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:51:58.351355 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348145 5463 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:51:58.351355 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348148 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:51:58.351355 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348150 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:51:58.351355 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348153 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:51:58.351355 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348155 5463 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:51:58.351355 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348158 5463 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:51:58.351355 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348160 5463 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:51:58.351355 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348162 5463 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:51:58.351355 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348165 5463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:51:58.351355 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348168 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:51:58.351355 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348170 5463 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:51:58.351355 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348173 5463 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:51:58.351355 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348176 5463 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:51:58.351355 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348179 5463 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:51:58.351355 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348182 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:51:58.351355 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348185 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:51:58.351893 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348202 5463 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:51:58.351893 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348205 5463 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:51:58.351893 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348207 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:51:58.351893 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348210 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:51:58.351893 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348213 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:51:58.351893 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348216 5463 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:51:58.351893 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348218 5463 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:51:58.351893 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348221 5463 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:51:58.351893 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348224 5463 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:51:58.351893 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348226 5463 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:51:58.351893 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348229 5463 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:51:58.351893 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348231 5463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:51:58.351893 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348235 5463 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:51:58.351893 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348238 5463 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:51:58.351893 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348722 5463 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:51:58.351893 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348728 5463 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:51:58.351893 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348732 5463 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:51:58.351893 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348735 5463 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:51:58.351893 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348737 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:51:58.352437 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348740 5463 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:51:58.352437 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348742 5463 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:51:58.352437 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348745 5463 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:51:58.352437 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348748 5463 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:51:58.352437 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348751 5463 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:51:58.352437 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348756 5463 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:51:58.352437 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348759 5463 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:51:58.352437 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348761 5463 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:51:58.352437 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348764 5463 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:51:58.352437 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348767 5463 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:51:58.352437 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348770 5463 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:51:58.352437 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348772 5463 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:51:58.352437 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348775 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:51:58.352437 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348777 5463 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:51:58.352437 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348780 5463 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:51:58.352437 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348783 5463 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:51:58.352437 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348785 5463 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:51:58.352437 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348788 5463 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:51:58.352437 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348790 5463 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:51:58.352437 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348793 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:51:58.353031 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348796 5463 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:51:58.353031 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348798 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:51:58.353031 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348801 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:51:58.353031 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348803 5463 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:51:58.353031 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348806 5463 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:51:58.353031 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348808 5463 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:51:58.353031 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348811 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:51:58.353031 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348813 5463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:51:58.353031 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348816 5463 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:51:58.353031 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348819 5463 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:51:58.353031 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348822 5463 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:51:58.353031 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348824 5463 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:51:58.353031 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348827 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:51:58.353031 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348829 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:51:58.353031 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348832 5463 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:51:58.353031 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348834 5463 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:51:58.353031 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348837 5463 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:51:58.353031 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348841 5463 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:51:58.353031 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348843 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:51:58.353817 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348846 5463 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:51:58.353817 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348849 5463 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:51:58.353817 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348851 5463 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:51:58.353817 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348854 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:51:58.353817 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348857 5463 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:51:58.353817 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348859 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:51:58.353817 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348862 5463 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:51:58.353817 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348865 5463 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:51:58.353817 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348867 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:51:58.353817 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348871 5463 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:51:58.353817 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348875 5463 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:51:58.353817 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348878 5463 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:51:58.353817 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348881 5463 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:51:58.353817 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348884 5463 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:51:58.353817 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348886 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:51:58.353817 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348889 5463 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:51:58.353817 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348891 5463 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:51:58.353817 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348894 5463 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:51:58.353817 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348897 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:51:58.354360 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348900 5463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:51:58.354360 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348902 5463 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:51:58.354360 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348905 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:51:58.354360 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348909 5463 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:51:58.354360 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348913 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:51:58.354360 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348916 5463 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:51:58.354360 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348918 5463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:51:58.354360 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348921 5463 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:51:58.354360 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348924 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:51:58.354360 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348926 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:51:58.354360 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348929 5463 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:51:58.354360 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348933 5463 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:51:58.354360 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348935 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:51:58.354360 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348938 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:51:58.354360 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348941 5463 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:51:58.354360 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348943 5463 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:51:58.354360 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348948 5463 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:51:58.354360 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348951 5463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:51:58.354360 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348953 5463 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:51:58.354831 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348956 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:51:58.354831 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348959 5463 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:51:58.354831 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348961 5463 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:51:58.354831 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348964 5463 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:51:58.354831 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348966 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:51:58.354831 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348969 5463 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:51:58.354831 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348972 5463 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:51:58.354831 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348974 5463 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:51:58.354831 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348977 5463 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:51:58.354831 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348979 5463 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:51:58.354831 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348982 5463 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:51:58.354831 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348984 5463 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:51:58.354831 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348987 5463 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:51:58.354831 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348989 5463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:51:58.354831 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348992 5463 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:51:58.354831 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348994 5463 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:51:58.354831 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348997 5463 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:51:58.354831 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.348999 5463 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:51:58.354831 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349002 5463 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:51:58.354831 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349004 5463 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349007 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349011 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349013 5463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349016 5463 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349104 5463 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349117 5463 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349125 5463 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349132 5463 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349138 5463 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349145 5463 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349151 5463 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349155 5463 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349158 5463 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349161 5463 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349164 5463 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349169 5463 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349172 5463 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349176 5463 flags.go:64] FLAG: --cgroup-root="" Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349178 5463 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349181 5463 flags.go:64] FLAG: --client-ca-file="" Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349184 5463 flags.go:64] FLAG: --cloud-provider="external" Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349217 5463 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 01:51:58.355335 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349224 5463 flags.go:64] FLAG: --cluster-domain="" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349227 5463 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349230 5463 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349234 5463 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349237 5463 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349242 5463 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349245 5463 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349248 5463 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349253 5463 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349256 5463 flags.go:64] FLAG: --contention-profiling="false" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349260 5463 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349263 5463 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349266 5463 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349269 5463 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349274 5463 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349277 5463 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349281 5463 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349283 5463 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349287 5463 flags.go:64] FLAG: --enable-server="true" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349290 5463 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349294 5463 flags.go:64] FLAG: --event-burst="100" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349298 5463 flags.go:64] FLAG: --event-qps="50" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349301 5463 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349304 5463 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 01:51:58.355871 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349307 5463 flags.go:64] FLAG: --eviction-hard="" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349311 5463 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349314 5463 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349318 5463 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349321 5463 flags.go:64] FLAG: --eviction-soft="" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349324 5463 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349327 5463 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349330 5463 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349334 5463 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349337 5463 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349340 5463 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349343 5463 flags.go:64] FLAG: --feature-gates="" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349347 5463 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349350 5463 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349353 5463 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349357 5463 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349360 5463 flags.go:64] FLAG: --healthz-port="10248" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349364 5463 flags.go:64] FLAG: --help="false" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349367 5463 flags.go:64] FLAG: --hostname-override="ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349370 5463 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349373 5463 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349376 5463 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349380 5463 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 01:51:58.356468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349384 5463 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349387 5463 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349390 5463 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349393 5463 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349396 5463 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349399 5463 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349402 5463 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349405 5463 flags.go:64] FLAG: --kube-reserved="" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349408 5463 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349411 5463 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349414 5463 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349417 5463 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349420 5463 flags.go:64] FLAG: --lock-file="" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349424 5463 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349427 5463 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349430 5463 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349435 5463 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349439 5463 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349442 5463 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349445 5463 flags.go:64] FLAG: --logging-format="text" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349448 5463 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349451 5463 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349454 5463 flags.go:64] FLAG: --manifest-url="" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349457 5463 flags.go:64] FLAG: --manifest-url-header="" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349462 5463 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 01:51:58.357017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349465 5463 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349468 5463 flags.go:64] FLAG: --max-pods="110" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349472 5463 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349475 5463 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349478 5463 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349481 5463 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349484 5463 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349487 5463 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349493 5463 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349499 5463 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349503 5463 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349506 5463 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349509 5463 flags.go:64] FLAG: --pod-cidr="" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349512 5463 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349515 5463 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349518 5463 flags.go:64] FLAG: --pods-per-core="0" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349521 5463 flags.go:64] FLAG: --port="10250" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349524 5463 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349527 5463 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-069bb454c0d19fd90" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349530 5463 flags.go:64] FLAG: --qos-reserved="" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349533 5463 flags.go:64] FLAG: --read-only-port="10255" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349537 5463 flags.go:64] FLAG: --register-node="true" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349540 5463 flags.go:64] FLAG: --register-with-taints="" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349544 5463 flags.go:64] FLAG: --registry-burst="10" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349547 5463 flags.go:64] FLAG: --registry-qps="5" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349549 5463 flags.go:64] FLAG: --reserved-cpus="" Nov 03 01:51:58.357657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349553 5463 flags.go:64] FLAG: --reserved-memory="" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349556 5463 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349559 5463 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349562 5463 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349565 5463 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349568 5463 flags.go:64] FLAG: --runonce="false" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349571 5463 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349574 5463 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349577 5463 flags.go:64] FLAG: --seccomp-default="false" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349581 5463 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349584 5463 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349587 5463 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349590 5463 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349593 5463 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349597 5463 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349600 5463 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349603 5463 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349606 5463 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349609 5463 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349612 5463 flags.go:64] FLAG: --system-cgroups="" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349615 5463 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349621 5463 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349624 5463 flags.go:64] FLAG: --tls-cert-file="" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349626 5463 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349630 5463 flags.go:64] FLAG: --tls-min-version="" Nov 03 01:51:58.358297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349653 5463 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349658 5463 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349661 5463 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349664 5463 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349668 5463 flags.go:64] FLAG: --v="2" Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349672 5463 flags.go:64] FLAG: --version="false" Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349677 5463 flags.go:64] FLAG: --vmodule="" Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349681 5463 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.349684 5463 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349832 5463 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349836 5463 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349839 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349842 5463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349845 5463 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349848 5463 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349851 5463 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349853 5463 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349857 5463 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349859 5463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349862 5463 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349865 5463 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:51:58.358916 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349870 5463 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:51:58.359548 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349873 5463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:51:58.359548 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349876 5463 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:51:58.359548 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349879 5463 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:51:58.359548 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349881 5463 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:51:58.359548 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349884 5463 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:51:58.359548 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349887 5463 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:51:58.359548 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349890 5463 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:51:58.359548 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349892 5463 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:51:58.359548 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349895 5463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:51:58.359548 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349898 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:51:58.359548 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349902 5463 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:51:58.359548 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349904 5463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:51:58.359548 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349907 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:51:58.359548 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349910 5463 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:51:58.359548 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349913 5463 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:51:58.359548 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349915 5463 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:51:58.359548 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349918 5463 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:51:58.359548 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349921 5463 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:51:58.359548 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349923 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:51:58.359548 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349926 5463 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349928 5463 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349931 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349934 5463 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349936 5463 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349939 5463 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349941 5463 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349944 5463 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349955 5463 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349960 5463 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349962 5463 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349965 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349970 5463 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349972 5463 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349975 5463 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349978 5463 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349981 5463 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349983 5463 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349986 5463 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349988 5463 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349991 5463 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:51:58.360142 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349994 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:51:58.360720 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.349997 5463 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:51:58.360720 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350003 5463 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:51:58.360720 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350006 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:51:58.360720 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350009 5463 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:51:58.360720 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350011 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:51:58.360720 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350014 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:51:58.360720 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350016 5463 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:51:58.360720 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350019 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:51:58.360720 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350021 5463 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:51:58.360720 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350024 5463 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:51:58.360720 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350027 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:51:58.360720 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350029 5463 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:51:58.360720 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350032 5463 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:51:58.360720 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350034 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:51:58.360720 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350037 5463 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:51:58.360720 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350040 5463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:51:58.360720 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350042 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:51:58.360720 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350045 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:51:58.360720 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350047 5463 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:51:58.361319 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350050 5463 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:51:58.361319 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350053 5463 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:51:58.361319 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350055 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:51:58.361319 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350059 5463 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:51:58.361319 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350062 5463 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:51:58.361319 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350065 5463 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:51:58.361319 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350068 5463 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:51:58.361319 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350070 5463 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:51:58.361319 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350073 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:51:58.361319 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350076 5463 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:51:58.361319 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350078 5463 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:51:58.361319 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350081 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:51:58.361319 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350083 5463 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:51:58.361319 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350086 5463 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:51:58.361319 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350090 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:51:58.361319 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350092 5463 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:51:58.361319 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350095 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:51:58.361319 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350098 5463 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:51:58.361319 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350100 5463 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:51:58.361854 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350103 5463 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:51:58.361854 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350106 5463 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:51:58.361854 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350108 5463 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:51:58.361854 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350111 5463 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:51:58.361854 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350115 5463 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:51:58.361854 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350119 5463 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:51:58.361854 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350123 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:51:58.361854 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350125 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:51:58.361854 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350128 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:51:58.361854 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350131 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:51:58.361854 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350134 5463 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:51:58.361854 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350136 5463 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:51:58.361854 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350139 5463 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:51:58.361854 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350143 5463 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:51:58.361854 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.350170 5463 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:51:58.362303 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350224 5463 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:51:58.362303 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350229 5463 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:51:58.362303 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350235 5463 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:51:58.362303 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350238 5463 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:51:58.362303 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350241 5463 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:51:58.362303 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350244 5463 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:51:58.362303 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350246 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:51:58.362303 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350249 5463 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:51:58.362303 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350251 5463 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:51:58.362303 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350254 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:51:58.362303 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350256 5463 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:51:58.362303 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350259 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:51:58.362303 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350263 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:51:58.362303 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350265 5463 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:51:58.362303 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350268 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:51:58.362303 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350270 5463 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:51:58.362303 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350273 5463 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:51:58.362303 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350276 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:51:58.362303 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350278 5463 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:51:58.362754 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350281 5463 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:51:58.362754 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350283 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:51:58.362754 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350287 5463 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:51:58.362754 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350289 5463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:51:58.362754 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350292 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:51:58.362754 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350294 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:51:58.362754 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350297 5463 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:51:58.362754 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350299 5463 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:51:58.362754 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350302 5463 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:51:58.362754 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350304 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:51:58.362754 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350307 5463 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:51:58.362754 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350310 5463 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:51:58.362754 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350312 5463 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:51:58.362754 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350315 5463 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:51:58.362754 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350318 5463 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:51:58.362754 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350322 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:51:58.362754 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350325 5463 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:51:58.362754 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350327 5463 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:51:58.362754 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350330 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:51:58.363380 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350333 5463 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:51:58.363380 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350335 5463 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:51:58.363380 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350338 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:51:58.363380 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350341 5463 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:51:58.363380 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350343 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:51:58.363380 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350346 5463 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:51:58.363380 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350350 5463 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:51:58.363380 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350353 5463 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:51:58.363380 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350356 5463 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:51:58.363380 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350359 5463 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:51:58.363380 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350361 5463 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:51:58.363380 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350365 5463 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:51:58.363380 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350370 5463 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:51:58.363380 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350373 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:51:58.363380 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350376 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:51:58.363380 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350378 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:51:58.363380 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350381 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:51:58.363380 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350385 5463 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:51:58.363380 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350387 5463 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:51:58.363938 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350390 5463 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:51:58.363938 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350393 5463 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:51:58.363938 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350397 5463 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:51:58.363938 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350400 5463 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:51:58.363938 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350403 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:51:58.363938 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350405 5463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:51:58.363938 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350409 5463 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:51:58.363938 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350411 5463 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:51:58.363938 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350414 5463 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:51:58.363938 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350418 5463 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:51:58.363938 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350420 5463 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:51:58.363938 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350423 5463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:51:58.363938 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350426 5463 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:51:58.363938 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350429 5463 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:51:58.363938 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350431 5463 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:51:58.363938 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350434 5463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:51:58.363938 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350437 5463 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:51:58.363938 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350439 5463 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:51:58.363938 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350442 5463 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:51:58.364444 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350446 5463 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:51:58.364444 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350448 5463 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:51:58.364444 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350451 5463 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:51:58.364444 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350454 5463 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:51:58.364444 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350456 5463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:51:58.364444 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350459 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:51:58.364444 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350461 5463 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:51:58.364444 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350464 5463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:51:58.364444 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350467 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:51:58.364444 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350469 5463 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:51:58.364444 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350472 5463 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:51:58.364444 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350474 5463 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:51:58.364444 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350477 5463 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:51:58.364444 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350479 5463 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:51:58.364444 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350482 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:51:58.364444 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350484 5463 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:51:58.364444 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350487 5463 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:51:58.364444 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350489 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:51:58.364444 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350492 5463 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:51:58.364444 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350494 5463 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:51:58.364918 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350497 5463 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:51:58.364918 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350499 5463 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:51:58.364918 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350503 5463 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:51:58.364918 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350506 5463 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:51:58.364918 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350509 5463 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:51:58.364918 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350512 5463 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:51:58.364918 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350514 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:51:58.364918 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350517 5463 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:51:58.364918 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350519 5463 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:51:58.364918 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.350522 5463 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:51:58.364918 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.355636 5463 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 01:51:58.364918 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.355653 5463 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 01:51:58.364918 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.355681 5463 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 01:51:58.364918 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.355687 5463 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 01:51:58.364918 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355743 5463 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:51:58.364918 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355761 5463 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:51:58.364918 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355766 5463 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:51:58.364918 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355769 5463 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:51:58.364918 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355772 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:51:58.364918 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355775 5463 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:51:58.365483 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355778 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:51:58.365483 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355781 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:51:58.365483 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355783 5463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:51:58.365483 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355786 5463 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:51:58.365483 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355789 5463 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:51:58.365483 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355791 5463 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:51:58.365483 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355794 5463 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:51:58.365483 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355797 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:51:58.365483 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355800 5463 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:51:58.365483 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355802 5463 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:51:58.365483 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355805 5463 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:51:58.365483 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355807 5463 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:51:58.365483 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355810 5463 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:51:58.365483 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355813 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:51:58.365483 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355816 5463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:51:58.365483 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355818 5463 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:51:58.365483 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355821 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:51:58.365483 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355824 5463 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:51:58.365483 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355828 5463 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:51:58.365483 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355831 5463 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:51:58.365946 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355834 5463 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:51:58.365946 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355836 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:51:58.365946 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355839 5463 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:51:58.365946 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355842 5463 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:51:58.365946 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355844 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:51:58.365946 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355847 5463 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:51:58.365946 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355850 5463 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:51:58.365946 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355854 5463 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:51:58.365946 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355857 5463 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:51:58.365946 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355860 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:51:58.365946 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355863 5463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:51:58.365946 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355866 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:51:58.365946 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355869 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:51:58.365946 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355872 5463 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:51:58.365946 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355875 5463 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:51:58.365946 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355878 5463 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:51:58.365946 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355880 5463 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:51:58.365946 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355883 5463 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:51:58.365946 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355886 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:51:58.366426 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355889 5463 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:51:58.366426 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355897 5463 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:51:58.366426 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355900 5463 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:51:58.366426 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355903 5463 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:51:58.366426 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355906 5463 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:51:58.366426 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355909 5463 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:51:58.366426 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355911 5463 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:51:58.366426 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355914 5463 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:51:58.366426 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355916 5463 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:51:58.366426 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355919 5463 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:51:58.366426 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355921 5463 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:51:58.366426 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355924 5463 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:51:58.366426 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355927 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:51:58.366426 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355929 5463 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:51:58.366426 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355932 5463 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:51:58.366426 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355936 5463 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:51:58.366426 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355939 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:51:58.366426 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355942 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:51:58.366426 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355945 5463 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:51:58.366878 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355948 5463 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:51:58.366878 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355951 5463 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:51:58.366878 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355954 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:51:58.366878 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355956 5463 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:51:58.366878 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355959 5463 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:51:58.366878 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355961 5463 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:51:58.366878 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355964 5463 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:51:58.366878 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355967 5463 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:51:58.366878 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355969 5463 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:51:58.366878 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355972 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:51:58.366878 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355975 5463 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:51:58.366878 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355978 5463 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:51:58.366878 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355980 5463 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:51:58.366878 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355983 5463 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:51:58.366878 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355985 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:51:58.366878 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355994 5463 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:51:58.366878 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355997 5463 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:51:58.366878 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.355999 5463 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:51:58.366878 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356002 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:51:58.366878 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356005 5463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:51:58.367369 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356008 5463 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:51:58.367369 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356010 5463 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:51:58.367369 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356013 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:51:58.367369 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356015 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:51:58.367369 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356018 5463 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:51:58.367369 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356021 5463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:51:58.367369 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356023 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:51:58.367369 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356026 5463 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:51:58.367369 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356029 5463 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:51:58.367369 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356031 5463 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:51:58.367369 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356034 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:51:58.367369 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356036 5463 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:51:58.367369 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356039 5463 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:51:58.367369 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356041 5463 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:51:58.367369 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356044 5463 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:51:58.367369 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356046 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:51:58.367369 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356049 5463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:51:58.367369 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356051 5463 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:51:58.367369 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356054 5463 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:51:58.367369 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356056 5463 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:51:58.367840 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356059 5463 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:51:58.367840 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356061 5463 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:51:58.367840 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.356088 5463 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:51:58.367840 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356231 5463 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:51:58.367840 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356236 5463 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:51:58.367840 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356239 5463 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:51:58.367840 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356242 5463 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:51:58.367840 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356245 5463 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:51:58.367840 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356249 5463 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:51:58.367840 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356252 5463 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:51:58.367840 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356255 5463 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:51:58.367840 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356257 5463 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:51:58.367840 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356260 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:51:58.367840 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356262 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:51:58.367840 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356265 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:51:58.367840 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356268 5463 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:51:58.367840 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356271 5463 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:51:58.368286 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356274 5463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:51:58.368286 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356276 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:51:58.368286 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356279 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:51:58.368286 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356281 5463 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:51:58.368286 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356284 5463 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:51:58.368286 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356286 5463 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:51:58.368286 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356289 5463 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:51:58.368286 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356291 5463 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:51:58.368286 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356294 5463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:51:58.368286 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356297 5463 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:51:58.368286 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356299 5463 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:51:58.368286 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356302 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:51:58.368286 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356304 5463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:51:58.368286 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356307 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:51:58.368286 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356310 5463 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:51:58.368286 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356312 5463 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:51:58.368286 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356315 5463 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:51:58.368286 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356317 5463 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:51:58.368286 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356320 5463 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:51:58.368745 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356324 5463 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:51:58.368745 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356328 5463 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:51:58.368745 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356333 5463 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:51:58.368745 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356336 5463 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:51:58.368745 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356339 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:51:58.368745 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356343 5463 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:51:58.368745 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356346 5463 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:51:58.368745 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356348 5463 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:51:58.368745 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356351 5463 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:51:58.368745 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356353 5463 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:51:58.368745 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356357 5463 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:51:58.368745 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356361 5463 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:51:58.368745 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356364 5463 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:51:58.368745 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356366 5463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:51:58.368745 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356369 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:51:58.368745 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356372 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:51:58.368745 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356374 5463 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:51:58.368745 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356377 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:51:58.369209 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356379 5463 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:51:58.369209 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356382 5463 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:51:58.369209 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356384 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:51:58.369209 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356387 5463 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:51:58.369209 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356389 5463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:51:58.369209 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356392 5463 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:51:58.369209 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356394 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:51:58.369209 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356397 5463 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:51:58.369209 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356399 5463 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:51:58.369209 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356402 5463 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:51:58.369209 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356404 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:51:58.369209 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356407 5463 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:51:58.369209 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356409 5463 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:51:58.369209 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356412 5463 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:51:58.369209 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356415 5463 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:51:58.369209 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356417 5463 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:51:58.369209 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356420 5463 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:51:58.369209 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356422 5463 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:51:58.369209 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356425 5463 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:51:58.369209 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356427 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:51:58.369771 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356430 5463 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:51:58.369771 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356433 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:51:58.369771 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356436 5463 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:51:58.369771 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356438 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:51:58.369771 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356441 5463 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:51:58.369771 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356443 5463 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:51:58.369771 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356446 5463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:51:58.369771 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356451 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:51:58.369771 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356453 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:51:58.369771 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356456 5463 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:51:58.369771 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356459 5463 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:51:58.369771 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356461 5463 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:51:58.369771 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356464 5463 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:51:58.369771 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356467 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:51:58.369771 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356469 5463 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:51:58.369771 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356472 5463 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:51:58.369771 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356475 5463 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:51:58.369771 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356478 5463 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:51:58.369771 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356480 5463 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:51:58.370304 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356483 5463 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:51:58.370304 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356486 5463 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:51:58.370304 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356489 5463 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:51:58.370304 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356491 5463 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:51:58.370304 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356494 5463 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:51:58.370304 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356496 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:51:58.370304 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356499 5463 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:51:58.370304 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356501 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:51:58.370304 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356504 5463 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:51:58.370304 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356506 5463 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:51:58.370304 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356509 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:51:58.370304 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356511 5463 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:51:58.370304 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356514 5463 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:51:58.370304 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356516 5463 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:51:58.370304 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356520 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:51:58.370304 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356523 5463 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:51:58.370717 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.356546 5463 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:51:58.370717 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356576 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:51:58.370717 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356581 5463 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:51:58.370717 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356584 5463 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:51:58.370717 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356586 5463 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:51:58.370717 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356593 5463 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:51:58.370717 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356595 5463 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:51:58.370717 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356598 5463 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:51:58.370717 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356601 5463 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:51:58.370717 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356603 5463 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:51:58.370717 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356606 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:51:58.370717 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356609 5463 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:51:58.370717 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356612 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:51:58.370717 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356615 5463 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:51:58.370717 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356617 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:51:58.370717 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356620 5463 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:51:58.371103 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356622 5463 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:51:58.371103 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356625 5463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:51:58.371103 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356627 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:51:58.371103 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356630 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:51:58.371103 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356633 5463 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:51:58.371103 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356635 5463 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:51:58.371103 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356637 5463 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:51:58.371103 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356640 5463 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:51:58.371103 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356643 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:51:58.371103 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356645 5463 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:51:58.371103 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356648 5463 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:51:58.371103 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356650 5463 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:51:58.371103 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356653 5463 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:51:58.371103 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356656 5463 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:51:58.371103 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356658 5463 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:51:58.371103 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356661 5463 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:51:58.371103 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356664 5463 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:51:58.371103 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356667 5463 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:51:58.371103 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356670 5463 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:51:58.371567 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356673 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:51:58.371567 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356675 5463 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:51:58.371567 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356679 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:51:58.371567 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356681 5463 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:51:58.371567 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356685 5463 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:51:58.371567 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356687 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:51:58.371567 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356690 5463 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:51:58.371567 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356692 5463 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:51:58.371567 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356695 5463 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:51:58.371567 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356698 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:51:58.371567 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356701 5463 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:51:58.371567 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356704 5463 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:51:58.371567 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356706 5463 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:51:58.371567 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356709 5463 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:51:58.371567 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356712 5463 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:51:58.371567 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356714 5463 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:51:58.371567 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356717 5463 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:51:58.371567 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356719 5463 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:51:58.371567 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356722 5463 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:51:58.371567 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356724 5463 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:51:58.372082 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356727 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:51:58.372082 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356730 5463 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:51:58.372082 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356732 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:51:58.372082 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356735 5463 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:51:58.372082 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356737 5463 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:51:58.372082 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356740 5463 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:51:58.372082 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356742 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:51:58.372082 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356745 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:51:58.372082 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356747 5463 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:51:58.372082 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356750 5463 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:51:58.372082 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356753 5463 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:51:58.372082 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356755 5463 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:51:58.372082 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356758 5463 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:51:58.372082 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356761 5463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:51:58.372082 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356764 5463 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:51:58.372082 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356767 5463 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:51:58.372082 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356770 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:51:58.372082 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356772 5463 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:51:58.372082 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356775 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:51:58.372082 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356778 5463 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:51:58.372579 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356781 5463 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:51:58.372579 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356783 5463 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:51:58.372579 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356786 5463 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:51:58.372579 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356788 5463 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:51:58.372579 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356792 5463 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:51:58.372579 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356797 5463 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:51:58.372579 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356800 5463 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:51:58.372579 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356803 5463 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:51:58.372579 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356806 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:51:58.372579 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356808 5463 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:51:58.372579 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356811 5463 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:51:58.372579 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356814 5463 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:51:58.372579 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356816 5463 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:51:58.372579 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356819 5463 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:51:58.372579 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356823 5463 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:51:58.372579 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356827 5463 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:51:58.372579 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356830 5463 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:51:58.372579 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356833 5463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:51:58.373009 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356835 5463 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:51:58.373009 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356838 5463 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:51:58.373009 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356840 5463 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:51:58.373009 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356843 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:51:58.373009 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356847 5463 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:51:58.373009 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356850 5463 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:51:58.373009 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356852 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:51:58.373009 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356855 5463 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:51:58.373009 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356859 5463 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:51:58.373009 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356862 5463 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:51:58.373009 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356864 5463 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:51:58.373009 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356867 5463 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:51:58.373009 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356869 5463 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:51:58.373009 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:51:58.356872 5463 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:51:58.373009 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.356924 5463 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 01:51:58.373009 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.357853 5463 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 01:51:58.373009 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.357914 5463 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 01:51:58.373009 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.358098 5463 server.go:1014] "Starting client certificate rotation" Nov 03 01:51:58.373009 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.358226 5463 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 01:51:58.373544 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.358266 5463 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 01:45:46 +0000 UTC" deadline="2028-06-01 20:10:07.557661261 +0000 UTC" Nov 03 01:51:58.373544 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.358302 5463 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13842h18m9.199361243s" Nov 03 01:51:58.373544 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.358602 5463 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 01:51:58.373544 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.359592 5463 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 01:51:58.373544 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.360974 5463 log.go:25] "Validated CRI v1 runtime API" Nov 03 01:51:58.373544 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.362982 5463 log.go:25] "Validated CRI v1 image API" Nov 03 01:51:58.373544 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.363715 5463 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 01:51:58.373544 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.365835 5463 fs.go:135] Filesystem UUIDs: map[02f3caf4-3feb-44bf-8ff2-7cdce39e5298:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f55849e7-412f-4376-9f52-1bfb7706adcb:/dev/nvme0n1p3] Nov 03 01:51:58.373765 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.365852 5463 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/20ebc1cce59a3f906431dc85e4f26347fa509a9d7dcdb0250c6c7d3a2177825e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/20ebc1cce59a3f906431dc85e4f26347fa509a9d7dcdb0250c6c7d3a2177825e/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25703b935a0c6e44085a51f4c26568296321b98ffb7cec9806cd5922db9785d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25703b935a0c6e44085a51f4c26568296321b98ffb7cec9806cd5922db9785d6/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3717a43ccb07013a2856316f7089ecbd346040aadf62076d5f53d359c368bfa6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3717a43ccb07013a2856316f7089ecbd346040aadf62076d5f53d359c368bfa6/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42222e9da675826f9421ad9fd953680996f440356720084df669922a8311d5a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42222e9da675826f9421ad9fd953680996f440356720084df669922a8311d5a7/userdata/shm major:0 minor:332 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d65afecf7a43d9f3f9c2bd7ebff7975998cfb0c9e385ace14617be4c88d6d2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d65afecf7a43d9f3f9c2bd7ebff7975998cfb0c9e385ace14617be4c88d6d2b/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6265a52693c0f05e96393ee46c5dbad14c43c910fc567a4db682ae4dccac0fd4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6265a52693c0f05e96393ee46c5dbad14c43c910fc567a4db682ae4dccac0fd4/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65f689eca5dcee2d78d686333cb851aaae8738109b9cfeeb80ce366bbdfe4e20/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65f689eca5dcee2d78d686333cb851aaae8738109b9cfeeb80ce366bbdfe4e20/userdata/shm major:0 minor:258 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/68fc421f756d8b4aece98a6c530ee369e80797f733640ef2c7cb13daf048f7a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/68fc421f756d8b4aece98a6c530ee369e80797f733640ef2c7cb13daf048f7a2/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8bcb594a4aea99826a085c1e5d37909a2d0102fd69b180356869231de34744e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8bcb594a4aea99826a085c1e5d37909a2d0102fd69b180356869231de34744e9/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e990765c3265bb643e07d980763c4be6bca8d5cd63e9a8fb727c140499c0c11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e990765c3265bb643e07d980763c4be6bca8d5cd63e9a8fb727c140499c0c11/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/950008f8e686ec9f369a11655f4dd77308b491c84e4912ddded048cc3bfbccee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/950008f8e686ec9f369a11655f4dd77308b491c84e4912ddded048cc3bfbccee/userdata/shm major:0 minor:405 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c54b0eeddd1cf0570b29b831e27d2d29c80f00ee78c0452d12575a29903b77f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c54b0eeddd1cf0570b29b831e27d2d29c80f00ee78c0452d12575a29903b77f/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7385d5f6115564b10316b7520473db1ca20a5b25d34d24c5d7614db270a9bd8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7385d5f6115564b10316b7520473db1ca20a5b25d34d24c5d7614db270a9bd8/userdata/shm major:0 minor:326 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf0fe3e961f1508c22d450e96ffb038537c15dbfb0ee3bb1eaf42182f0410dbc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf0fe3e961f1508c22d450e96ffb038537c15dbfb0ee3bb1eaf42182f0410dbc/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c265f9fcbf01ea35d2596ad8d6f2eaa46f5c09b097a945653dd36e26ad541db4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c265f9fcbf01ea35d2596ad8d6f2eaa46f5c09b097a945653dd36e26ad541db4/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d78d72da05776d0a39a73dac13f3863f84b23c3eef6fbde0ab624f44e7d46ed0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d78d72da05776d0a39a73dac13f3863f84b23c3eef6fbde0ab624f44e7d46ed0/userdata/shm major:0 minor:304 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbdf0dca6157a139cbf3d23144097c985eed5ed94f46bd069f92d5794d24af84/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbdf0dca6157a139cbf3d23144097c985eed5ed94f46bd069f92d5794d24af84/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5c8a3898daa81e3d35949131b00ac85309812bc32d5e0a6ffa725258ab4d22b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5c8a3898daa81e3d35949131b00ac85309812bc32d5e0a6ffa725258ab4d22b/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e749d1c30334c484eccbc9087758c8f4ed5ec8903fb35c78c933da96f9227ec8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e749d1c30334c484eccbc9087758c8f4ed5ec8903fb35c78c933da96f9227ec8/userdata/shm major:0 minor:337 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7d1ff2e711fb13ffbc06c310106d4b6cb873440fad8ccdeb3b7f30fdd3172df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7d1ff2e711fb13ffbc06c310106d4b6cb873440fad8ccdeb3b7f30fdd3172df/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee588b93a46b68acc7542cebd9b40c450fe5db9aa2ff8eadae86fa933f64a411/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee588b93a46b68acc7542cebd9b40c450fe5db9aa2ff8eadae86fa933f64a411/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f221678002fffd8191846f3e7395f648c0f6ca625f7b8c8605a7276967d46808/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f221678002fffd8191846f3e7395f648c0f6ca625f7b8c8605a7276967d46808/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0155a1d8-3fb6-48da-9319-c65514404dd1/volumes/kubernetes.io~projected/kube-api-access-v9clj:{mountpoint:/var/lib/kubelet/pods/0155a1d8-3fb6-48da-9319-c65514404dd1/volumes/kubernetes.io~projected/kube-api-access-v9clj major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14044133-e020-4562-84ec-9c09ce6e6e3c/volumes/kubernetes.io~projected/kube-api-access-nqpwg:{mountpoint:/var/lib/kubelet/pods/14044133-e020-4562-84ec-9c09ce6e6e3c/volumes/kubernetes.io~projected/kube-api-access-nqpwg major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14044133-e020-4562-84ec-9c09ce6e6e3c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/14044133-e020-4562-84ec-9c09ce6e6e3c/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1553379d-e8f5-4668-b524-13f5b93d0811/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1553379d-e8f5-4668-b524-13f5b93d0811/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1553379d-e8f5-4668-b524-13f5b93d0811/volumes/kubernetes.io~projected/kube-api-access-s7fh8:{mountpoint:/var/lib/kubelet/pods/1553379d-e8f5-4668-b524-13f5b93d0811/volumes/kubernetes.io~projected/kube-api-access-s7fh8 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1553379d-e8f5-4668-b524-13f5b93d0811/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1553379d-e8f5-4668-b524-13f5b93d0811/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/307c837d-2477-4460-9606-46efe63be297/volumes/kubernetes.io~projected/kube-api-access-k759x:{mountpoint:/var/lib/kubelet/pods/307c837d-2477-4460-9606-46efe63be297/volumes/kubernetes.io~projected/kube-api-access-k759x major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3799f150-114a-4ed4-a955-e121465de3ab/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3799f150-114a-4ed4-a955-e121465de3ab/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37dd5ea7-75ae-4414-8c9a-90fe106cfd48/volumes/kubernetes.io~projected/kube-api-access-srgxc:{mountpoint:/var/lib/kubelet/pods/37dd5ea7-75ae-4414-8c9a-90fe106cfd48/volumes/kubernetes.io~projected/kube-api-access-srgxc major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37dd5ea7-75ae-4414-8c9a-90fe106cfd48/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/37dd5ea7-75ae-4414-8c9a-90fe106cfd48/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37dd5ea7-75ae-4414-8c9a-90fe106cfd48/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/37dd5ea7-75ae-4414-8c9a-90fe106cfd48/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61b2893f-ece1-496b-b25d-56dc1fbb1f89/volumes/kubernetes.io~projected/kube-api-access-9nnhh:{mountpoint:/var/lib/kubelet/pods/61b2893f-ece1-496b-b25d-56dc1fbb1f89/volumes/kubernetes.io~projected/kube-api-access-9nnhh major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61b2893f-ece1-496b-b25d-56dc1fbb1f89/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/61b2893f-ece1-496b-b25d-56dc1fbb1f89/volumes/kubernetes.io~secret/metrics-certs major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ffde31b-d09a-4739-9bd4-abe007571006/volumes/kubernetes.io~projected/kube-api-access-gzprl:{mountpoint:/var/lib/kubelet/pods/6ffde31b-d09a-4739-9bd4-abe007571006/volumes/kubernetes.io~projected/kube-api-access-gzprl major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95977c23-899f-4892-8b57-c0e47ea6ec82/volumes/kubernetes.io~projected/kube-api-access-8c7dg:{mountpoint:/var/lib/kubelet/pods/95977c23-899f-4892-8b57-c0e47ea6ec82/volumes/kubernetes.io~projected/kube-api-access-8c7dg major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~empty-dir/config-out major:0 minor:390 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~projected/kube-api-access-sv668:{mountpoint:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~projected/kube-api-access-sv668 major:0 minor:404 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~projected/tls-assets major:0 minor:399 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/config major:0 minor:395 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:394 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:403 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:402 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/web-config major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7b495c8-cb0d-4107-b99e-3f11833410d6/volumes/kubernetes.io~projected/kube-api-access-lzpf7:{mountpoint:/var/lib/kubelet/pods/a7b495c8-cb0d-4107-b99e-3f11833410d6/volumes/kubernetes.io~projected/kube-api-access-lzpf7 major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7b495c8-cb0d-4107-b99e-3f11833410d6/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a7b495c8-cb0d-4107-b99e-3f11833410d6/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7b495c8-cb0d-4107-b99e-3f11833410d6/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/a7b495c8-cb0d-4107-b99e-3f11833410d6/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aafc0e0e-1526-4a94-a8c0-163e80562ea8/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/aafc0e0e-1526-4a94-a8c0-163e80562ea8/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aafc0e0e-1526-4a94-a8c0-163e80562ea8/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/aafc0e0e-1526-4a94-a8c0-163e80562ea8/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aafc0e0e-1526-4a94-a8c0-163e80562ea8/volumes/kubernetes.io~projected/kube-api-access-pkbxw:{mountpoint:/var/lib/kubelet/pods/aafc0e0e-1526-4a94-a8c0-163e80562ea8/volumes/kubernetes.io~projected/kube-api-access-pkbxw major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89/volumes/kubernetes.io~projected/kube-api-access-xltsq:{mountpoint:/var/lib/kubelet/pods/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89/volumes/kubernetes.io~projected/kube-api-access-xltsq major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c546c8d0-b721-45f9-b528-2e3490e84bef/volumes/kubernetes.io~projected/kube-api-access-mgm6s:{mountpoint:/var/lib/kubelet/pods/c546c8d0-b721-45f9-b528-2e3490e84bef/volumes/kubernetes.io~projected/kube-api-access-mgm6s major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd268007-d02d-44bc-8ad8-ad15135c1ae2/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/cd268007-d02d-44bc-8ad8-ad15135c1ae2/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf67907b-cfa3-40ea-812d-4ea034671442/volumes/kubernetes.io~projected/kube-api-access-xnj6b:{mountpoint:/var/lib/kubelet/pods/cf67907b-cfa3-40ea-812d-4ea034671442/volumes/kubernetes.io~projected/kube-api-access-xnj6b major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf67907b-cfa3-40ea-812d-4ea034671442/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/cf67907b-cfa3-40ea-812d-4ea034671442/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/edf584ed-41f3-4fff-9e58-261b51e70b32/volumes/kubernetes.io~projected/kube-api-access-4q7cl:{mountpoint:/var/lib/kubelet/pods/edf584ed-41f3-4fff-9e58-261b51e70b32/volumes/kubernetes.io~projected/kube-api-access-4q7cl major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/edf584ed-41f3-4fff-9e58-261b51e70b32/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/edf584ed-41f3-4fff-9e58-261b51e70b32/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/edf584ed-41f3-4fff-9e58-261b51e70b32/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/edf584ed-41f3-4fff-9e58-261b51e70b32/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f15be6a7-766d-4eae-8e57-3a34850ca41b/volumes/kubernetes.io~projected/kube-api-access-52pmv:{mountpoint:/var/lib/kubelet/pods/f15be6a7-766d-4eae-8e57-3a34850ca41b/volumes/kubernetes.io~projected/kube-api-access-52pmv major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f15be6a7-766d-4eae-8e57-3a34850ca41b/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/f15be6a7-766d-4eae-8e57-3a34850ca41b/volumes/kubernetes.io~secret/metrics-tls major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~projected/kube-api-access-2mbln:{mountpoint:/var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~projected/kube-api-access-2mbln major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcd16c62-1c32-448a-9f84-45fb913f26ba/volumes/kubernetes.io~projected/kube-api-access-97v2w:{mountpoint:/var/lib/kubelet/pods/fcd16c62-1c32-448a-9f84-45fb913f26ba/volumes/kubernetes.io~projected/kube-api-access-97v2w major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcd16c62-1c32-448a-9f84-45fb913f26ba/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/fcd16c62-1c32-448a-9f84-45fb913f26ba/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729/volumes/kubernetes.io~projected/kube-api-access-rqfmb:{mountpoint:/var/lib/kubelet/pods/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729/volumes/kubernetes.io~projected/kube-api-access-rqfmb major:0 minor:57 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/52846ffddbc28e28fe04f00d9508774139c120f33769b14abad23316831d11bb/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b2c160da8988e6dbf2431717fd8b16de18d9b50f917135db60c5803159c5e4d6/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b8342f3968c916df77aeab89919a89b1516876c50b284c8320a6ea76073f3401/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/c07497dd85b4cc4d0a563999e96baf161d283d967f26f4fcc99b162311ed728b/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/987defb13a42563aa178fef72cf9e318f87ca30aa65426e864782622b1751998/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a2df04529a59fe4ae88b8575adc238de94050a848262e3c8ef881dc81ebd9d5b/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/92459620d4a855b79f2fc8934e4d6e045c75a686e55c7c460f38e2488b06f718/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/02807a979bcf1eb8b1b0b4c287224d0da9418e9e05b43be7911293c633b8aea0/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/e56badaf3eb9a8a80b34c448df518b528fe8128ce29fd952d4af25e109bd4b32/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/1fea73af551d8c1fa9abf190bab84e052a31a515feadfa5c2e769f7075501747/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/0ad97d73084a01b0fb43456e1e02f2ed81856b367960407042c57d77909829b0/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/200f854be79231860e8bcef9a3479e7c76025afc27b462cc14a0dbf1484cd47f/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/f8932c3704d15cc580a0ebc404fa302c964de5d12a98ffb7eb9cdb7c5b2c2980/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/a9f384d597ee574efe45e02fd1eb235ca3986ce9d7cb89459ea3c2c498487aa5/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/7834cf33b080d1463714f9ce3aafce84ca009d84c2365d9391f2873d7125de29/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/0404207f43123be60ac6852c63290630953c6815713a5d7c4087e1ee2cf489a3/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/857b597ec3a730bf91b965c9df368a6dc22fee09ec197d5d5e861522357cd846/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/697ec4c3c05eef326c43c9af870d1922e3226ccb5e2d622f661b431ce4e2bc28/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/containers/storage/overlay/3c086e4337825d484fb64c6f619beb57ab36ca9984eddd6390b9fff1434ba1dc/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e44ccc0b3048baad2f89abc8b45feb0daf91ae5b91bfa66e872bdbbe2c61bb9a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f06c2c60c585557c78faddedf030f872fbf30232c367c45b69bd82a04fecf253/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/80514643160aa3dc051403f41976ee0eecf209caf1c10c1a90d13f65e163022c/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/3ea678fd2d04462bb0ef7283817f5565fa0fef82e02341abe401780084647b4a/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/821345f1608390237709271a08af3dca80c8ddb2dd4986090f3a492580bb6ef3/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/a5d0a72d1e83512f31ae3731364974df7ef8d2fe56dea826758426120a23a263/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/d6f8b0b4a0c2aada12dae1fbf12866fa72b322331de33517f663f4a2999aa0f8/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/c4afe29ddece0fcb992c717fad63f1d10d905c144b1d67f8725b0e3fe1151973/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/d37777b3a6315e06b0d4e379b02c1f49c946197b5799b9924ffda14d3a45d9b4/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/756a97ad047106375216cfaf4b3056f1680c597859aa533c849004503ba07660/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/4662f4c8956ff6de974d409ddcda7aaf4aaaa51245922da1dc60df58b6e784fc/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/e79c3e985d0550d1aab575469fb3b1b6bd74c7e026e266b2ba881560c41ef6fb/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/5e41c1238cc2e52342c65bc0813d9af9325a3e7fec8faab43405527a74d84d0a/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/43ee22e3f2a4c870a7f179078750f82039a30ba5e0bbc857b1f47e7d5cbf3931/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/374b0628073b76bbd3a69be75522de3f5e745ccfa3311cbc08b2aacb2db7370f/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/dedddf230197e4f9abe18b1b70df7e5bf8d0424b2922a8e2b43c0dcb2f1f902b/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/7ebb8a89b455b15bb8d21a8a66d1c334d8dabc4be788556953339f7affdc288c/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/223c140d9e7cd0997ddab519990c79fb28c551faba191fcddd3cdf00d0a23061/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/9e65d40bcfa8b302324d1cf7f8594ff4d88c5fc25c305fb3d6d7e2baa31420b1/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/4925e8ba84dddd998e3d3c6bfb7d097ff198414e04372effa75aa08e6e5fa087/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/575b1e5d09cccf11dbe4a64c3b2b5a054093e40e1ea4a3f357cd96536b554def/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/c661639654b14ff1d60a3af6ca9d7e05114939f8fe24e5bce45b54cfb321f7ca/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/1d9d8f278d6179b822af39fa91c43e140082a158df7f54e2c453dd7e2c396780/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/55f5b0c86c699219b4580f7a68da66f1e64b40acc760264ce91c974f2a74a64b/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/b04b3163d9e74a0fadd1b6fe44ce2fc6205c2e268989049d224727b16b7b8ea7/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/668752595a3fc48c426a95dfae761391e6ac734f7546c759186fe3c230483356/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/cc885fe5102707c9dc0888c987699d1ae2b1c871ffb8106ba49669379300222b/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/d4a58616915534ac1b11e7df654ae083b09e3ddc931f34c07363e79a15dacb47/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/3cfe3b0f945f7479332d7b347ae819f7d8cb36241595b5d9a297db0c7346e59e/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/d0343a8ff6b05284f176b0645b42a1175f449dbd8299eb2f0f5ee5b63c2dfbb4/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/c78d7c91ae4d0c98fd9b26514be4abb00161ecd9b58a7606394beb8a5198219c/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/b31e07da1f2a68dd5c33ba702d231e82f91e0021a3dcf83c55101b3586038ac4/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/ebc24d19d98950820b06e337abcde632ade4cee71fb308d66d348b249ae0232f/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/516595ed45319279a90805d40b642c967188e3207c4a4a3701fedf9d97129e5d/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/d50e7476c9e5ff0dd61d93a45cc2582653a104e8bf40d78478b45dcdc15fc33d/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/1eac2f694b0941054f8a2f5ed46df85eccbd9382dee85c69f22b05ddc8fe2b1d/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/85d60202f44fdc87419fb5717ee41fb6b797c16d8e094310ca348bf1633bbf7e/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/2b7a72e7f4148371817f9fcd128663ab105fe856a79730c4feb6a31393401218/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/4d6ab87ad78d8571604ef0fbf955a0e8d5918fc6554718011f47fbcd493e24ea/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/bc8aa33f56da89875cab76f0b3532e79a3a5ae02debabddc2cb437375e058378/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/b68e809474f7902e3625feb78edda0d7ba82b8c3ebdedd102dae1425c79aa8cd/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/62e14b78f86450a0a695edfaf73e516522f36bf915b193ffdc05a09bb3bb07e6/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/4e6f28c649f1475bb26124de28b6a6aa5cd5ca28099ee3df75e9ca07094d378b/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/8f766bce83963a9ac839f2884bfb6e76973811da2def44ebd242da45cc8f53c3/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/5ad01c0a436cd29c880efed7ce292162e23ec9d9e0ddc3da1a3ea0b0eaa80447/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/c2d57b3e4ea4eb3f0db777e32dabc8b02bb8b8ef7d46d7b944630a18602db3f0/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/6bb1ded8d6950cc10533148fd96ddd24db6f91a5f478f7f423b2fd9d22c54c05/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/f306a124bd1ffcc7e441b3eb8e09a8df75ac8f6102209f0f51fcf3a8b2939bf0/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/452e01ffc87b496a462b420babf4d0d1021833303ce3e78c4434719f5cbfb01e/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 01:51:58.374674 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.374243 5463 manager.go:217] Machine: {Timestamp:2026-11-03 01:51:58.373754739 +0000 UTC m=+0.057863886 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3143330 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29146fb66cdab1687ed1462a4fdb15 SystemUUID:ec29146f-b66c-dab1-687e-d1462a4fdb15 BootID:6a290036-ec23-486b-b277-45db85a32cfe Filesystems:[{Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f15be6a7-766d-4eae-8e57-3a34850ca41b/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:397 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f221678002fffd8191846f3e7395f648c0f6ca625f7b8c8605a7276967d46808/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14044133-e020-4562-84ec-9c09ce6e6e3c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/14044133-e020-4562-84ec-9c09ce6e6e3c/volumes/kubernetes.io~projected/kube-api-access-nqpwg DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:403 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cf67907b-cfa3-40ea-812d-4ea034671442/volumes/kubernetes.io~projected/kube-api-access-xnj6b DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcd16c62-1c32-448a-9f84-45fb913f26ba/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:324 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89/volumes/kubernetes.io~projected/kube-api-access-xltsq DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cf67907b-cfa3-40ea-812d-4ea034671442/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d78d72da05776d0a39a73dac13f3863f84b23c3eef6fbde0ab624f44e7d46ed0/userdata/shm DeviceMajor:0 DeviceMinor:304 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:398 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd268007-d02d-44bc-8ad8-ad15135c1ae2/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:328 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e749d1c30334c484eccbc9087758c8f4ed5ec8903fb35c78c933da96f9227ec8/userdata/shm DeviceMajor:0 DeviceMinor:337 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:390 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c265f9fcbf01ea35d2596ad8d6f2eaa46f5c09b097a945653dd36e26ad541db4/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ffde31b-d09a-4739-9bd4-abe007571006/volumes/kubernetes.io~projected/kube-api-access-gzprl DeviceMajor:0 DeviceMinor:334 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:396 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:400 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:398 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729/volumes/kubernetes.io~projected/kube-api-access-rqfmb DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/edf584ed-41f3-4fff-9e58-261b51e70b32/volumes/kubernetes.io~projected/kube-api-access-4q7cl DeviceMajor:0 DeviceMinor:255 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/37dd5ea7-75ae-4414-8c9a-90fe106cfd48/volumes/kubernetes.io~projected/kube-api-access-srgxc DeviceMajor:0 DeviceMinor:260 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/aafc0e0e-1526-4a94-a8c0-163e80562ea8/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:395 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1553379d-e8f5-4668-b524-13f5b93d0811/volumes/kubernetes.io~projected/kube-api-access-s7fh8 DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7d1ff2e711fb13ffbc06c310106d4b6cb873440fad8ccdeb3b7f30fdd3172df/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c54b0eeddd1cf0570b29b831e27d2d29c80f00ee78c0452d12575a29903b77f/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf0fe3e961f1508c22d450e96ffb038537c15dbfb0ee3bb1eaf42182f0410dbc/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a7b495c8-cb0d-4107-b99e-3f11833410d6/volumes/kubernetes.io~projected/kube-api-access-lzpf7 DeviceMajor:0 DeviceMinor:253 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7385d5f6115564b10316b7520473db1ca20a5b25d34d24c5d7614db270a9bd8/userdata/shm DeviceMajor:0 DeviceMinor:326 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/edf584ed-41f3-4fff-9e58-261b51e70b32/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:246 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65f689eca5dcee2d78d686333cb851aaae8738109b9cfeeb80ce366bbdfe4e20/userdata/shm DeviceMajor:0 DeviceMinor:258 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aafc0e0e-1526-4a94-a8c0-163e80562ea8/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aafc0e0e-1526-4a94-a8c0-163e80562ea8/volumes/kubernetes.io~projected/kube-api-access-pkbxw DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/edf584ed-41f3-4fff-9e58-261b51e70b32/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/37dd5ea7-75ae-4414-8c9a-90fe106cfd48/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:256 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8bcb594a4aea99826a085c1e5d37909a2d0102fd69b180356869231de34744e9/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:299 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~projected/kube-api-access-2mbln DeviceMajor:0 DeviceMinor:303 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a7b495c8-cb0d-4107-b99e-3f11833410d6/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:254 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1553379d-e8f5-4668-b524-13f5b93d0811/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee588b93a46b68acc7542cebd9b40c450fe5db9aa2ff8eadae86fa933f64a411/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3717a43ccb07013a2856316f7089ecbd346040aadf62076d5f53d359c368bfa6/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6265a52693c0f05e96393ee46c5dbad14c43c910fc567a4db682ae4dccac0fd4/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-181 DeviceMajor:0 DeviceMinor:181 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:291 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbdf0dca6157a139cbf3d23144097c985eed5ed94f46bd069f92d5794d24af84/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/950008f8e686ec9f369a11655f4dd77308b491c84e4912ddded048cc3bfbccee/userdata/shm DeviceMajor:0 DeviceMinor:405 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c546c8d0-b721-45f9-b528-2e3490e84bef/volumes/kubernetes.io~projected/kube-api-access-mgm6s DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1553379d-e8f5-4668-b524-13f5b93d0811/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:402 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/307c837d-2477-4460-9606-46efe63be297/volumes/kubernetes.io~projected/kube-api-access-k759x DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/20ebc1cce59a3f906431dc85e4f26347fa509a9d7dcdb0250c6c7d3a2177825e/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5c8a3898daa81e3d35949131b00ac85309812bc32d5e0a6ffa725258ab4d22b/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25703b935a0c6e44085a51f4c26568296321b98ffb7cec9806cd5922db9785d6/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/61b2893f-ece1-496b-b25d-56dc1fbb1f89/volumes/kubernetes.io~projected/kube-api-access-9nnhh DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e990765c3265bb643e07d980763c4be6bca8d5cd63e9a8fb727c140499c0c11/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d65afecf7a43d9f3f9c2bd7ebff7975998cfb0c9e385ace14617be4c88d6d2b/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/61b2893f-ece1-496b-b25d-56dc1fbb1f89/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:329 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/68fc421f756d8b4aece98a6c530ee369e80797f733640ef2c7cb13daf048f7a2/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:300 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:301 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f15be6a7-766d-4eae-8e57-3a34850ca41b/volumes/kubernetes.io~projected/kube-api-access-52pmv DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a7b495c8-cb0d-4107-b99e-3f11833410d6/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:252 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:302 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:399 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/95977c23-899f-4892-8b57-c0e47ea6ec82/volumes/kubernetes.io~projected/kube-api-access-8c7dg DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3799f150-114a-4ed4-a955-e121465de3ab/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~projected/kube-api-access-sv668 DeviceMajor:0 DeviceMinor:404 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/37dd5ea7-75ae-4414-8c9a-90fe106cfd48/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:257 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fad00740-ad1a-4623-ba88-2b2319c14266/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:292 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42222e9da675826f9421ad9fd953680996f440356720084df669922a8311d5a7/userdata/shm DeviceMajor:0 DeviceMinor:332 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0155a1d8-3fb6-48da-9319-c65514404dd1/volumes/kubernetes.io~projected/kube-api-access-v9clj DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fcd16c62-1c32-448a-9f84-45fb913f26ba/volumes/kubernetes.io~projected/kube-api-access-97v2w DeviceMajor:0 DeviceMinor:325 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:401 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:394 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:42222e9da675826 MacAddress:8a:83:55:1a:ba:b9 Speed:10000 Mtu:8901} {Name:65f689eca5dcee2 MacAddress:0e:8b:2b:3b:b9:d5 Speed:10000 Mtu:8901} {Name:68fc421f756d8b4 MacAddress:5a:31:49:68:93:14 Speed:10000 Mtu:8901} {Name:8bcb594a4aea998 MacAddress:ca:1d:5b:60:8f:f6 Speed:10000 Mtu:8901} {Name:950008f8e686ec9 MacAddress:ee:34:82:8d:fe:18 Speed:10000 Mtu:8901} {Name:a7385d5f6115564 MacAddress:92:d3:7f:78:9e:e4 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:f3:8c:47:75 Speed:0 Mtu:9001} {Name:br-int MacAddress:42:a8:2a:ac:b3:0e Speed:0 Mtu:8901} {Name:c265f9fcbf01ea3 MacAddress:92:61:4e:ef:cb:1e Speed:10000 Mtu:8901} {Name:d78d72da05776d0 MacAddress:8e:d6:d2:97:66:80 Speed:10000 Mtu:8901} {Name:dbdf0dca6157a13 MacAddress:6e:67:b3:12:98:c7 Speed:10000 Mtu:8901} {Name:e5c8a3898daa81e MacAddress:32:a2:d4:2c:42:be Speed:10000 Mtu:8901} {Name:e749d1c30334c48 MacAddress:76:5f:48:4f:df:ff Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:f3:8c:47:75 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:e6:51:09:cd:64:67 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1e:a1:31:02:0c:cb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 01:51:58.374917 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.374678 5463 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 01:51:58.374917 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.374722 5463 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 01:51:58.375032 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.375005 5463 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 01:51:58.375185 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.375034 5463 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-69.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 01:51:58.375185 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.375198 5463 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 01:51:58.375301 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.375208 5463 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 01:51:58.375301 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.375219 5463 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 01:51:58.375301 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.375231 5463 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 01:51:58.375410 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.375314 5463 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 01:51:58.375514 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.375501 5463 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 01:51:58.375628 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.375618 5463 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 01:51:58.375663 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.375653 5463 kubelet.go:493] "Attempting to sync node with API server" Nov 03 01:51:58.375663 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.375662 5463 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 01:51:58.375717 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.375672 5463 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 01:51:58.375717 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.375680 5463 kubelet.go:405] "Adding apiserver pod source" Nov 03 01:51:58.375717 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.375688 5463 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 01:51:58.376471 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.376451 5463 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 01:51:58.376471 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.376469 5463 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 01:51:58.376916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.376892 5463 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 01:51:58.391153 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.391130 5463 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 01:51:58.391153 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.391146 5463 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 01:51:58.391309 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.391279 5463 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 01:51:58.391502 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.391486 5463 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 01:51:58.391562 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.391527 5463 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 01:51:58.391653 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.391642 5463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 01:51:58.391708 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.391662 5463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 01:51:58.391708 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.391673 5463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 01:51:58.391708 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.391684 5463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 01:51:58.391708 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.391693 5463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 01:51:58.391708 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.391702 5463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 01:51:58.391936 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.391712 5463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 01:51:58.391936 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.391722 5463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 01:51:58.391936 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.391734 5463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 01:51:58.391936 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.391743 5463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 01:51:58.391936 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.391756 5463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 01:51:58.391936 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.391771 5463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 01:51:58.391936 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.391807 5463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 01:51:58.391936 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.391816 5463 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 01:51:58.396087 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.395389 5463 server.go:1263] "Started kubelet" Nov 03 01:51:58.396087 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.395947 5463 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 01:51:58.395891 ip-10-0-137-69 systemd[1]: Started Kubernetes Kubelet. Nov 03 01:51:58.397263 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.397102 5463 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 01:51:58.397263 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.397182 5463 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 01:51:58.397548 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.397518 5463 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 01:51:58.401880 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.400972 5463 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 01:51:58.401880 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.400992 5463 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 01:51:58.401880 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.401040 5463 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 01:45:47 +0000 UTC" deadline="2028-07-31 16:36:05.941353506 +0000 UTC" Nov 03 01:51:58.401880 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.401058 5463 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15278h44m7.540298659s" Nov 03 01:51:58.401880 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.401061 5463 server.go:317] "Adding debug handlers to kubelet server" Nov 03 01:51:58.401880 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.401070 5463 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 01:51:58.401880 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.401079 5463 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 01:51:58.401880 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:51:58.401107 5463 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-69.us-west-2.compute.internal\" not found" Nov 03 01:51:58.401880 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.401176 5463 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 01:51:58.403316 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403279 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37dd5ea7-75ae-4414-8c9a-90fe106cfd48" volumeName="kubernetes.io/projected/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-api-access-srgxc" seLinuxMountContext="" Nov 03 01:51:58.403316 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403317 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95977c23-899f-4892-8b57-c0e47ea6ec82" volumeName="kubernetes.io/configmap/95977c23-899f-4892-8b57-c0e47ea6ec82-serviceca" seLinuxMountContext="" Nov 03 01:51:58.403427 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403326 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7b495c8-cb0d-4107-b99e-3f11833410d6" volumeName="kubernetes.io/secret/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 01:51:58.403427 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403335 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf67907b-cfa3-40ea-812d-4ea034671442" volumeName="kubernetes.io/configmap/cf67907b-cfa3-40ea-812d-4ea034671442-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 01:51:58.403427 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403343 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f15be6a7-766d-4eae-8e57-3a34850ca41b" volumeName="kubernetes.io/configmap/f15be6a7-766d-4eae-8e57-3a34850ca41b-config-volume" seLinuxMountContext="" Nov 03 01:51:58.403427 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403351 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a134e24f-d1ad-467a-914e-0ff35ca0177b" volumeName="kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-metrics-client-ca" seLinuxMountContext="" Nov 03 01:51:58.403427 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403364 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a134e24f-d1ad-467a-914e-0ff35ca0177b" volumeName="kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-grpc-tls" seLinuxMountContext="" Nov 03 01:51:58.403427 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403376 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf67907b-cfa3-40ea-812d-4ea034671442" volumeName="kubernetes.io/secret/cf67907b-cfa3-40ea-812d-4ea034671442-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 01:51:58.403427 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403388 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad00740-ad1a-4623-ba88-2b2319c14266" volumeName="kubernetes.io/projected/fad00740-ad1a-4623-ba88-2b2319c14266-kube-api-access-2mbln" seLinuxMountContext="" Nov 03 01:51:58.403427 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403398 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcd16c62-1c32-448a-9f84-45fb913f26ba" volumeName="kubernetes.io/secret/fcd16c62-1c32-448a-9f84-45fb913f26ba-monitoring-plugin-cert" seLinuxMountContext="" Nov 03 01:51:58.403427 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403415 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a134e24f-d1ad-467a-914e-0ff35ca0177b" volumeName="kubernetes.io/projected/a134e24f-d1ad-467a-914e-0ff35ca0177b-kube-api-access-sv668" seLinuxMountContext="" Nov 03 01:51:58.403427 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403422 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcd16c62-1c32-448a-9f84-45fb913f26ba" volumeName="kubernetes.io/projected/fcd16c62-1c32-448a-9f84-45fb913f26ba-kube-api-access-97v2w" seLinuxMountContext="" Nov 03 01:51:58.403427 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403430 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1553379d-e8f5-4668-b524-13f5b93d0811" volumeName="kubernetes.io/configmap/1553379d-e8f5-4668-b524-13f5b93d0811-ovnkube-config" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403437 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a134e24f-d1ad-467a-914e-0ff35ca0177b" volumeName="kubernetes.io/empty-dir/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-db" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403446 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7b495c8-cb0d-4107-b99e-3f11833410d6" volumeName="kubernetes.io/secret/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-tls" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403457 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be3a1b21-68d9-4a7b-b1e4-30669c2dbb89" volumeName="kubernetes.io/projected/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89-kube-api-access-xltsq" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403469 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c546c8d0-b721-45f9-b528-2e3490e84bef" volumeName="kubernetes.io/configmap/c546c8d0-b721-45f9-b528-2e3490e84bef-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403498 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad00740-ad1a-4623-ba88-2b2319c14266" volumeName="kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403512 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad00740-ad1a-4623-ba88-2b2319c14266" volumeName="kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-tls" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403526 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0155a1d8-3fb6-48da-9319-c65514404dd1" volumeName="kubernetes.io/projected/0155a1d8-3fb6-48da-9319-c65514404dd1-kube-api-access-v9clj" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403538 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37dd5ea7-75ae-4414-8c9a-90fe106cfd48" volumeName="kubernetes.io/empty-dir/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-volume-directive-shadow" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403551 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ffde31b-d09a-4739-9bd4-abe007571006" volumeName="kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403562 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a134e24f-d1ad-467a-914e-0ff35ca0177b" volumeName="kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403575 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7b495c8-cb0d-4107-b99e-3f11833410d6" volumeName="kubernetes.io/configmap/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403588 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37dd5ea7-75ae-4414-8c9a-90fe106cfd48" volumeName="kubernetes.io/configmap/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-metrics-client-ca" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403624 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf67907b-cfa3-40ea-812d-4ea034671442" volumeName="kubernetes.io/empty-dir/cf67907b-cfa3-40ea-812d-4ea034671442-data-volume" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403648 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14044133-e020-4562-84ec-9c09ce6e6e3c" volumeName="kubernetes.io/secret/14044133-e020-4562-84ec-9c09ce6e6e3c-cert" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403658 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="307c837d-2477-4460-9606-46efe63be297" volumeName="kubernetes.io/configmap/307c837d-2477-4460-9606-46efe63be297-cni-binary-copy" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403672 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3799f150-114a-4ed4-a955-e121465de3ab" volumeName="kubernetes.io/configmap/3799f150-114a-4ed4-a955-e121465de3ab-konnectivity-ca" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403694 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37dd5ea7-75ae-4414-8c9a-90fe106cfd48" volumeName="kubernetes.io/secret/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-state-metrics-tls" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403707 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37dd5ea7-75ae-4414-8c9a-90fe106cfd48" volumeName="kubernetes.io/secret/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403715 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a134e24f-d1ad-467a-914e-0ff35ca0177b" volumeName="kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403732 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a134e24f-d1ad-467a-914e-0ff35ca0177b" volumeName="kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403747 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a134e24f-d1ad-467a-914e-0ff35ca0177b" volumeName="kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-metrics-client-certs" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403754 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="307c837d-2477-4460-9606-46efe63be297" volumeName="kubernetes.io/configmap/307c837d-2477-4460-9606-46efe63be297-multus-daemon-config" seLinuxMountContext="" Nov 03 01:51:58.403766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403771 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37dd5ea7-75ae-4414-8c9a-90fe106cfd48" volumeName="kubernetes.io/configmap/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403779 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a134e24f-d1ad-467a-914e-0ff35ca0177b" volumeName="kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403786 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a134e24f-d1ad-467a-914e-0ff35ca0177b" volumeName="kubernetes.io/empty-dir/a134e24f-d1ad-467a-914e-0ff35ca0177b-config-out" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403798 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a134e24f-d1ad-467a-914e-0ff35ca0177b" volumeName="kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403806 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7b495c8-cb0d-4107-b99e-3f11833410d6" volumeName="kubernetes.io/configmap/a7b495c8-cb0d-4107-b99e-3f11833410d6-metrics-client-ca" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403813 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aafc0e0e-1526-4a94-a8c0-163e80562ea8" volumeName="kubernetes.io/empty-dir/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-tuned" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403827 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aafc0e0e-1526-4a94-a8c0-163e80562ea8" volumeName="kubernetes.io/empty-dir/aafc0e0e-1526-4a94-a8c0-163e80562ea8-tmp" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403834 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14044133-e020-4562-84ec-9c09ce6e6e3c" volumeName="kubernetes.io/projected/14044133-e020-4562-84ec-9c09ce6e6e3c-kube-api-access-nqpwg" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403843 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1553379d-e8f5-4668-b524-13f5b93d0811" volumeName="kubernetes.io/configmap/1553379d-e8f5-4668-b524-13f5b93d0811-env-overrides" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403854 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1553379d-e8f5-4668-b524-13f5b93d0811" volumeName="kubernetes.io/secret/1553379d-e8f5-4668-b524-13f5b93d0811-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403874 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="307c837d-2477-4460-9606-46efe63be297" volumeName="kubernetes.io/projected/307c837d-2477-4460-9606-46efe63be297-kube-api-access-k759x" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403882 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aafc0e0e-1526-4a94-a8c0-163e80562ea8" volumeName="kubernetes.io/projected/aafc0e0e-1526-4a94-a8c0-163e80562ea8-kube-api-access-pkbxw" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403890 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf67907b-cfa3-40ea-812d-4ea034671442" volumeName="kubernetes.io/projected/cf67907b-cfa3-40ea-812d-4ea034671442-kube-api-access-xnj6b" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403897 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f15be6a7-766d-4eae-8e57-3a34850ca41b" volumeName="kubernetes.io/empty-dir/f15be6a7-766d-4eae-8e57-3a34850ca41b-tmp-dir" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403908 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f15be6a7-766d-4eae-8e57-3a34850ca41b" volumeName="kubernetes.io/secret/f15be6a7-766d-4eae-8e57-3a34850ca41b-metrics-tls" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403918 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3799f150-114a-4ed4-a955-e121465de3ab" volumeName="kubernetes.io/secret/3799f150-114a-4ed4-a955-e121465de3ab-agent-certs" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403935 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61b2893f-ece1-496b-b25d-56dc1fbb1f89" volumeName="kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403945 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95977c23-899f-4892-8b57-c0e47ea6ec82" volumeName="kubernetes.io/projected/95977c23-899f-4892-8b57-c0e47ea6ec82-kube-api-access-8c7dg" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403957 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c546c8d0-b721-45f9-b528-2e3490e84bef" volumeName="kubernetes.io/projected/c546c8d0-b721-45f9-b528-2e3490e84bef-kube-api-access-mgm6s" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403969 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edf584ed-41f3-4fff-9e58-261b51e70b32" volumeName="kubernetes.io/projected/edf584ed-41f3-4fff-9e58-261b51e70b32-kube-api-access-4q7cl" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403980 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad00740-ad1a-4623-ba88-2b2319c14266" volumeName="kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.403998 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad00740-ad1a-4623-ba88-2b2319c14266" volumeName="kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-grpc-tls" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404011 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1553379d-e8f5-4668-b524-13f5b93d0811" volumeName="kubernetes.io/configmap/1553379d-e8f5-4668-b524-13f5b93d0811-ovnkube-script-lib" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404018 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a134e24f-d1ad-467a-914e-0ff35ca0177b" volumeName="kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404032 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c546c8d0-b721-45f9-b528-2e3490e84bef" volumeName="kubernetes.io/configmap/c546c8d0-b721-45f9-b528-2e3490e84bef-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404040 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad00740-ad1a-4623-ba88-2b2319c14266" volumeName="kubernetes.io/configmap/fad00740-ad1a-4623-ba88-2b2319c14266-metrics-client-ca" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404047 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1553379d-e8f5-4668-b524-13f5b93d0811" volumeName="kubernetes.io/projected/1553379d-e8f5-4668-b524-13f5b93d0811-kube-api-access-s7fh8" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404064 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a134e24f-d1ad-467a-914e-0ff35ca0177b" volumeName="kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-web-config" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404076 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd268007-d02d-44bc-8ad8-ad15135c1ae2" volumeName="kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404087 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edf584ed-41f3-4fff-9e58-261b51e70b32" volumeName="kubernetes.io/secret/edf584ed-41f3-4fff-9e58-261b51e70b32-openshift-state-metrics-tls" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404099 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad00740-ad1a-4623-ba88-2b2319c14266" volumeName="kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404106 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729" volumeName="kubernetes.io/projected/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729-kube-api-access-rqfmb" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404114 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61b2893f-ece1-496b-b25d-56dc1fbb1f89" volumeName="kubernetes.io/projected/61b2893f-ece1-496b-b25d-56dc1fbb1f89-kube-api-access-9nnhh" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404121 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a134e24f-d1ad-467a-914e-0ff35ca0177b" volumeName="kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-config" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404129 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7b495c8-cb0d-4107-b99e-3f11833410d6" volumeName="kubernetes.io/projected/a7b495c8-cb0d-4107-b99e-3f11833410d6-kube-api-access-lzpf7" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404136 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f15be6a7-766d-4eae-8e57-3a34850ca41b" volumeName="kubernetes.io/projected/f15be6a7-766d-4eae-8e57-3a34850ca41b-kube-api-access-52pmv" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404150 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729" volumeName="kubernetes.io/configmap/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729-iptables-alerter-script" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404158 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a134e24f-d1ad-467a-914e-0ff35ca0177b" volumeName="kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404171 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a134e24f-d1ad-467a-914e-0ff35ca0177b" volumeName="kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404181 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a134e24f-d1ad-467a-914e-0ff35ca0177b" volumeName="kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404202 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7b495c8-cb0d-4107-b99e-3f11833410d6" volumeName="kubernetes.io/empty-dir/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-textfile" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404221 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c546c8d0-b721-45f9-b528-2e3490e84bef" volumeName="kubernetes.io/configmap/c546c8d0-b721-45f9-b528-2e3490e84bef-cni-binary-copy" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404234 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad00740-ad1a-4623-ba88-2b2319c14266" volumeName="kubernetes.io/empty-dir/fad00740-ad1a-4623-ba88-2b2319c14266-tmp" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404245 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad00740-ad1a-4623-ba88-2b2319c14266" volumeName="kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404255 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a134e24f-d1ad-467a-914e-0ff35ca0177b" volumeName="kubernetes.io/projected/a134e24f-d1ad-467a-914e-0ff35ca0177b-tls-assets" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404267 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a134e24f-d1ad-467a-914e-0ff35ca0177b" volumeName="kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404282 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a134e24f-d1ad-467a-914e-0ff35ca0177b" volumeName="kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404292 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be3a1b21-68d9-4a7b-b1e4-30669c2dbb89" volumeName="kubernetes.io/empty-dir/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89-tmp-dir" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404303 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edf584ed-41f3-4fff-9e58-261b51e70b32" volumeName="kubernetes.io/configmap/edf584ed-41f3-4fff-9e58-261b51e70b32-metrics-client-ca" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404314 5463 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edf584ed-41f3-4fff-9e58-261b51e70b32" volumeName="kubernetes.io/secret/edf584ed-41f3-4fff-9e58-261b51e70b32-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404338 5463 reconstruct.go:97] "Volume reconstruction finished" Nov 03 01:51:58.404668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404345 5463 reconciler.go:29] "Reconciler: start to sync state" Nov 03 01:51:58.406945 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404790 5463 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 01:51:58.406945 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404810 5463 factory.go:55] Registering systemd factory Nov 03 01:51:58.406945 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.404822 5463 factory.go:223] Registration of the systemd container factory successfully Nov 03 01:51:58.406945 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.405135 5463 factory.go:153] Registering CRI-O factory Nov 03 01:51:58.406945 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.405154 5463 factory.go:223] Registration of the crio container factory successfully Nov 03 01:51:58.406945 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.405182 5463 factory.go:103] Registering Raw factory Nov 03 01:51:58.406945 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.405209 5463 manager.go:1199] Started watching for new ooms in manager Nov 03 01:51:58.406945 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.405558 5463 manager.go:319] Starting recovery of all containers Nov 03 01:51:58.424389 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.422955 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 01:51:58.426434 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.426412 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 01:51:58.433253 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.433227 5463 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 01:51:58.434909 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.434889 5463 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 01:51:58.435007 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.434913 5463 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 01:51:58.435007 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.434934 5463 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 01:51:58.435007 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:51:58.434971 5463 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 01:51:58.458981 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.458957 5463 generic.go:356] "Generic (PLEG): container finished" podID="a7b495c8-cb0d-4107-b99e-3f11833410d6" containerID="ba101f3f99d72517c6159e89c3df35b6a55bce333f0c654c18d232f6a830dc28" exitCode=0 Nov 03 01:51:58.465083 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.465056 5463 generic.go:356] "Generic (PLEG): container finished" podID="c546c8d0-b721-45f9-b528-2e3490e84bef" containerID="42ef1a25c50a3a583fd792e14512d164fdba4ed2e347a5c53cae457147ab75d4" exitCode=0 Nov 03 01:51:58.465083 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.465082 5463 generic.go:356] "Generic (PLEG): container finished" podID="c546c8d0-b721-45f9-b528-2e3490e84bef" containerID="343898d6bfc82942c83a76de21107128cd398eb46da2ff864ad5623c44eade53" exitCode=0 Nov 03 01:51:58.465248 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.465095 5463 generic.go:356] "Generic (PLEG): container finished" podID="c546c8d0-b721-45f9-b528-2e3490e84bef" containerID="591b3be2ce73b502fb2ee127538da7250ba2cc3ca944305e3f3281667db849bc" exitCode=0 Nov 03 01:51:58.465248 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.465104 5463 generic.go:356] "Generic (PLEG): container finished" podID="c546c8d0-b721-45f9-b528-2e3490e84bef" containerID="6718ff900de311a6af79299c38f5c55dadd99060859185e1da930154253a6d1e" exitCode=0 Nov 03 01:51:58.465248 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.465113 5463 generic.go:356] "Generic (PLEG): container finished" podID="c546c8d0-b721-45f9-b528-2e3490e84bef" containerID="875ab6116e69326a51a568818deb2b1f276f42d7f4883675ca27314eaeecada4" exitCode=0 Nov 03 01:51:58.465248 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.465121 5463 generic.go:356] "Generic (PLEG): container finished" podID="c546c8d0-b721-45f9-b528-2e3490e84bef" containerID="d4e1f327fcddc36b33052bd02575e5ab3e7e1acf3c14ad2b76019013b4aadff4" exitCode=0 Nov 03 01:51:58.467125 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.467100 5463 generic.go:356] "Generic (PLEG): container finished" podID="f7cc04d950edb247291a58fdd1584226" containerID="476b28bee5b67e7ab089e8d85b053ac02144b062aff9dc8845d3361aa17deb90" exitCode=0 Nov 03 01:51:58.470161 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.470137 5463 generic.go:356] "Generic (PLEG): container finished" podID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerID="e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066" exitCode=0 Nov 03 01:51:58.487131 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.487111 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 01:51:58.489746 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.489716 5463 generic.go:356] "Generic (PLEG): container finished" podID="1553379d-e8f5-4668-b524-13f5b93d0811" containerID="1ede88e3c22d1fa7301d722db396f695fab38a3ea3b8537e4e129bfff3585824" exitCode=1 Nov 03 01:51:58.501393 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:51:58.501373 5463 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-69.us-west-2.compute.internal\" not found" Nov 03 01:51:58.505544 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.505529 5463 manager.go:324] Recovery completed Nov 03 01:51:58.524866 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.524822 5463 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:51:58.526908 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.526892 5463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:51:58.526964 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.526925 5463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:51:58.526964 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.526939 5463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:51:58.529512 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.529496 5463 cpu_manager.go:226] "Starting" policy="none" Nov 03 01:51:58.529512 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.529509 5463 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 01:51:58.529622 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.529526 5463 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 01:51:58.529656 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.529510 5463 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:51:58.529656 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.529632 5463 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 01:51:58.529656 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.529641 5463 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 01:51:58.529656 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.529657 5463 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 01:51:58.529759 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.529663 5463 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 01:51:58.529759 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.529668 5463 policy_none.go:50] "Start" Nov 03 01:51:58.529759 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.529676 5463 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 01:51:58.529759 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.529687 5463 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 01:51:58.530514 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.530498 5463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:51:58.530584 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.530526 5463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:51:58.530584 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.530539 5463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:51:58.530822 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.530806 5463 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 01:51:58.530861 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.530828 5463 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 01:51:58.530861 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.530835 5463 policy_none.go:44] "Start" Nov 03 01:51:58.535078 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:51:58.535057 5463 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 01:51:58.535359 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.535327 5463 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 01:51:58.536780 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.536763 5463 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:51:58.538102 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.538089 5463 manager.go:341] "Starting Device Plugin manager" Nov 03 01:51:58.538180 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:51:58.538114 5463 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 01:51:58.538180 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.538122 5463 server.go:87] "Starting device plugin registration server" Nov 03 01:51:58.538421 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.538410 5463 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 01:51:58.538903 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.538422 5463 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 01:51:58.539007 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.538988 5463 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 01:51:58.539130 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.539090 5463 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 01:51:58.539240 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.539134 5463 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 01:51:58.539240 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.539141 5463 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 01:51:58.539352 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.539206 5463 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T01:51:58.539123773Z","UUID":"49bb9a82-155f-4971-be20-3d42613271c8","Handler":null,"Name":"","Endpoint":""} Nov 03 01:51:58.540453 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:51:58.540436 5463 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 01:51:58.540522 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:51:58.540476 5463 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-69.us-west-2.compute.internal\" not found" Nov 03 01:51:58.540840 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.540821 5463 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 01:51:58.540911 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.540850 5463 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 01:51:58.587304 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.587272 5463 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:51:58.635473 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.635445 5463 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:51:58.635587 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.635571 5463 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 01:51:58.639261 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.639247 5463 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:51:58.640232 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.640218 5463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:51:58.640316 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.640247 5463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:51:58.640316 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.640262 5463 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:51:58.640316 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.640312 5463 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:58.687361 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.687339 5463 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:51:58.736034 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.735980 5463 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal"] Nov 03 01:51:58.736635 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.736620 5463 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:58.736710 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.736627 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" event={"ID":"f7cc04d950edb247291a58fdd1584226","Type":"ContainerStarted","Data":"41d5f34f0ba4f4d748c91942a600dfc381ca36c23995f63352de22e2766ab3ed"} Nov 03 01:51:58.736710 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.736685 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" event={"ID":"f7cc04d950edb247291a58fdd1584226","Type":"ContainerDied","Data":"476b28bee5b67e7ab089e8d85b053ac02144b062aff9dc8845d3361aa17deb90"} Nov 03 01:51:58.736822 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.736721 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" event={"ID":"f7cc04d950edb247291a58fdd1584226","Type":"ContainerStarted","Data":"f221678002fffd8191846f3e7395f648c0f6ca625f7b8c8605a7276967d46808"} Nov 03 01:51:58.736822 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.736729 5463 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:58.736822 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.736736 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal" event={"ID":"d43bad4e10a0a43dceae949566bb34d7","Type":"ContainerStarted","Data":"41a5aa6b885db006de39631c03c729a3718649259d829f8120b830667982f472"} Nov 03 01:51:58.736822 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.736748 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal" event={"ID":"d43bad4e10a0a43dceae949566bb34d7","Type":"ContainerStarted","Data":"9c54b0eeddd1cf0570b29b831e27d2d29c80f00ee78c0452d12575a29903b77f"} Nov 03 01:51:58.748787 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.748768 5463 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:58.748878 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.748831 5463 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:58.797062 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.797015 5463 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:51:58.797062 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:51:58.797054 5463 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:58.797338 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.797324 5463 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:51:58.797395 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:51:58.797358 5463 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:58.836842 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.836822 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7cc04d950edb247291a58fdd1584226-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal\" (UID: \"f7cc04d950edb247291a58fdd1584226\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:58.836911 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.836889 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7cc04d950edb247291a58fdd1584226-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal\" (UID: \"f7cc04d950edb247291a58fdd1584226\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:58.836955 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.836930 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d43bad4e10a0a43dceae949566bb34d7-config\") pod \"kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal\" (UID: \"d43bad4e10a0a43dceae949566bb34d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:58.937612 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.937582 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7cc04d950edb247291a58fdd1584226-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal\" (UID: \"f7cc04d950edb247291a58fdd1584226\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:58.937612 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.937600 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7cc04d950edb247291a58fdd1584226-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal\" (UID: \"f7cc04d950edb247291a58fdd1584226\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:58.937780 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.937636 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7cc04d950edb247291a58fdd1584226-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal\" (UID: \"f7cc04d950edb247291a58fdd1584226\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:58.937780 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.937664 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d43bad4e10a0a43dceae949566bb34d7-config\") pod \"kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal\" (UID: \"d43bad4e10a0a43dceae949566bb34d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:58.937780 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.937712 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d43bad4e10a0a43dceae949566bb34d7-config\") pod \"kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal\" (UID: \"d43bad4e10a0a43dceae949566bb34d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:58.937780 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:58.937723 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7cc04d950edb247291a58fdd1584226-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal\" (UID: \"f7cc04d950edb247291a58fdd1584226\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:59.376748 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.376714 5463 apiserver.go:52] "Watching apiserver" Nov 03 01:51:59.481005 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.480976 5463 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 01:51:59.481573 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.481552 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-98ckh","openshift-dns/dns-default-nbqt6","openshift-dns/node-resolver-skjrl","openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq","openshift-cluster-node-tuning-operator/tuned-jvq7p","openshift-insights/insights-runtime-extractor-mdxfm","kube-system/konnectivity-agent-bx24c","kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-qmlx7","openshift-ingress-canary/ingress-canary-8sstb","openshift-network-diagnostics/network-check-target-rb2h2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal","openshift-monitoring/prometheus-k8s-0","openshift-multus/multus-additional-cni-plugins-c6nzj","openshift-ovn-kubernetes/ovnkube-node-mbmwx","openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb","openshift-network-operator/iptables-alerter-47qlq","openshift-monitoring/node-exporter-w7h88","openshift-multus/multus-g8nt7","openshift-monitoring/thanos-querier-54564794f6-z44tk","kube-system/global-pull-secret-syncer-r2z98","openshift-image-registry/node-ca-ggvl5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc"] Nov 03 01:51:59.495536 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.495514 5463 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:59.495671 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.495630 5463 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:59.503423 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.503408 5463 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 01:51:59.535530 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.535512 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2mmqt\"" Nov 03 01:51:59.535530 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.535512 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 01:51:59.535813 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.535797 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 01:51:59.535864 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.535821 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 01:51:59.536130 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.536116 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f7c55\"" Nov 03 01:51:59.536399 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.536380 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 01:51:59.536479 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.536405 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:51:59.536516 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.536494 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 01:51:59.536568 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.536549 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:51:59.536568 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.536551 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 01:51:59.536714 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.536639 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bqcwb\"" Nov 03 01:51:59.536881 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.536866 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 01:51:59.536954 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.536931 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zxsvp\"" Nov 03 01:51:59.536954 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.536950 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 01:51:59.537055 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.536980 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-btxgv\"" Nov 03 01:51:59.537134 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.537114 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jwmm7\"" Nov 03 01:51:59.537261 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.537159 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 01:51:59.537261 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.537160 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 01:51:59.537261 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.537227 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 01:51:59.537261 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.537226 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 01:51:59.537261 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.537256 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 01:51:59.537448 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.537293 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hgghc\"" Nov 03 01:51:59.537448 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.537343 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 01:51:59.537552 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.537541 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 01:51:59.538081 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.538070 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7cgvv\"" Nov 03 01:51:59.538331 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.538316 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 01:51:59.538331 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.538328 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 01:51:59.538415 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.538367 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 01:51:59.538577 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.538565 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 01:51:59.539069 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.539056 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 01:51:59.539123 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.539079 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 01:51:59.539123 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.539090 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 01:51:59.539646 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.539634 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 01:51:59.539695 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.539634 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pc8bh\"" Nov 03 01:51:59.539865 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.539850 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 01:51:59.540144 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.540127 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 01:51:59.540275 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.540164 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wpc48\"" Nov 03 01:51:59.540275 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.540173 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 01:51:59.540275 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.540216 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zq5dk\"" Nov 03 01:51:59.540470 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.540459 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 01:51:59.540636 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.540622 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5qdp4\"" Nov 03 01:51:59.540915 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.540901 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 01:51:59.540956 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.540947 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 01:51:59.541126 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.541111 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 01:51:59.541458 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.541444 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 01:51:59.541543 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.541526 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:59.541606 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.541553 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f15be6a7-766d-4eae-8e57-3a34850ca41b-metrics-tls\") pod \"dns-default-nbqt6\" (UID: \"f15be6a7-766d-4eae-8e57-3a34850ca41b\") " pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:59.541606 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.541572 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.541606 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.541596 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.541750 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.541620 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.541750 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.541646 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c7dg\" (UniqueName: \"kubernetes.io/projected/95977c23-899f-4892-8b57-c0e47ea6ec82-kube-api-access-8c7dg\") pod \"node-ca-ggvl5\" (UID: \"95977c23-899f-4892-8b57-c0e47ea6ec82\") " pod="openshift-image-registry/node-ca-ggvl5" Nov 03 01:51:59.541750 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.541653 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 01:51:59.541750 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.541667 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 01:51:59.541750 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.541730 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-run-multus-certs\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.541750 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.541732 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 01:51:59.542063 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.541764 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-etc-kubernetes\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.542063 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.541776 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jwnvd\"" Nov 03 01:51:59.542063 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.541815 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnj6b\" (UniqueName: \"kubernetes.io/projected/cf67907b-cfa3-40ea-812d-4ea034671442-kube-api-access-xnj6b\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:59.542063 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.541866 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q7cl\" (UniqueName: \"kubernetes.io/projected/edf584ed-41f3-4fff-9e58-261b51e70b32-kube-api-access-4q7cl\") pod \"openshift-state-metrics-7bd7489957-rdhbb\" (UID: \"edf584ed-41f3-4fff-9e58-261b51e70b32\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:59.542063 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.541900 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fad00740-ad1a-4623-ba88-2b2319c14266-tmp\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:59.542063 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.541927 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xltsq\" (UniqueName: \"kubernetes.io/projected/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89-kube-api-access-xltsq\") pod \"node-resolver-skjrl\" (UID: \"be3a1b21-68d9-4a7b-b1e4-30669c2dbb89\") " pod="openshift-dns/node-resolver-skjrl" Nov 03 01:51:59.542063 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.541961 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-systemd\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.542063 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.541985 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-run\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.542063 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.542022 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fad00740-ad1a-4623-ba88-2b2319c14266-tmp\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:59.542563 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.542545 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs\") pod \"network-metrics-daemon-qmlx7\" (UID: \"61b2893f-ece1-496b-b25d-56dc1fbb1f89\") " pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:59.542614 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.542583 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cf67907b-cfa3-40ea-812d-4ea034671442-crio-socket\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:59.542669 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.542611 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf67907b-cfa3-40ea-812d-4ea034671442-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:59.542669 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.542635 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89-tmp-dir\") pod \"node-resolver-skjrl\" (UID: \"be3a1b21-68d9-4a7b-b1e4-30669c2dbb89\") " pod="openshift-dns/node-resolver-skjrl" Nov 03 01:51:59.542669 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.542662 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:51:59.542830 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.542688 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.542830 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.542713 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/307c837d-2477-4460-9606-46efe63be297-multus-daemon-config\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.542830 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.542737 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-run-systemd\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.542830 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.542759 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89-tmp-dir\") pod \"node-resolver-skjrl\" (UID: \"be3a1b21-68d9-4a7b-b1e4-30669c2dbb89\") " pod="openshift-dns/node-resolver-skjrl" Nov 03 01:51:59.542830 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.542771 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-var-lib-openvswitch\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.542830 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.542795 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b2893f-ece1-496b-b25d-56dc1fbb1f89-metrics-certs\") pod \"network-metrics-daemon-qmlx7\" (UID: \"61b2893f-ece1-496b-b25d-56dc1fbb1f89\") " pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:59.542830 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.542812 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-node-log\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.542830 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.542821 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf67907b-cfa3-40ea-812d-4ea034671442-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:59.543144 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.542856 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-log-socket\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.543144 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.542904 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-cni-bin\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.543144 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.542918 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/307c837d-2477-4460-9606-46efe63be297-multus-daemon-config\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.543144 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.542934 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c546c8d0-b721-45f9-b528-2e3490e84bef-cnibin\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:51:59.543144 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.542959 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c546c8d0-b721-45f9-b528-2e3490e84bef-cni-binary-copy\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:51:59.543144 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.542977 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97v2w\" (UniqueName: \"kubernetes.io/projected/fcd16c62-1c32-448a-9f84-45fb913f26ba-kube-api-access-97v2w\") pod \"monitoring-plugin-77f4f76594-98ckh\" (UID: \"fcd16c62-1c32-448a-9f84-45fb913f26ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-98ckh" Nov 03 01:51:59.543144 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.543011 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-config\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.543144 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.543134 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-systemd-units\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.543491 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.543167 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-run-openvswitch\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.543491 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.543220 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c546c8d0-b721-45f9-b528-2e3490e84bef-cni-binary-copy\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:51:59.543491 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.543216 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52pmv\" (UniqueName: \"kubernetes.io/projected/f15be6a7-766d-4eae-8e57-3a34850ca41b-kube-api-access-52pmv\") pod \"dns-default-nbqt6\" (UID: \"f15be6a7-766d-4eae-8e57-3a34850ca41b\") " pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:59.543491 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.543265 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-var-lib-kubelet\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.543491 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.543291 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf67907b-cfa3-40ea-812d-4ea034671442-data-volume\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:59.543491 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.543320 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fad00740-ad1a-4623-ba88-2b2319c14266-metrics-client-ca\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:59.543491 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.543391 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf67907b-cfa3-40ea-812d-4ea034671442-data-volume\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:59.543893 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.543424 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fcd16c62-1c32-448a-9f84-45fb913f26ba-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-98ckh\" (UID: \"fcd16c62-1c32-448a-9f84-45fb913f26ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-98ckh" Nov 03 01:51:59.543941 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.543909 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.543941 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.543926 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-run-netns\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.544027 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.543944 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqpwg\" (UniqueName: \"kubernetes.io/projected/14044133-e020-4562-84ec-9c09ce6e6e3c-kube-api-access-nqpwg\") pod \"ingress-canary-8sstb\" (UID: \"14044133-e020-4562-84ec-9c09ce6e6e3c\") " pod="openshift-ingress-canary/ingress-canary-8sstb" Nov 03 01:51:59.544027 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.543959 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-sysconfig\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.544027 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.543974 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k759x\" (UniqueName: \"kubernetes.io/projected/307c837d-2477-4460-9606-46efe63be297-kube-api-access-k759x\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.544027 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.543998 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:59.544027 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544018 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mbln\" (UniqueName: \"kubernetes.io/projected/fad00740-ad1a-4623-ba88-2b2319c14266-kube-api-access-2mbln\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:59.544255 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544079 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3799f150-114a-4ed4-a955-e121465de3ab-konnectivity-ca\") pod \"konnectivity-agent-bx24c\" (UID: \"3799f150-114a-4ed4-a955-e121465de3ab\") " pod="kube-system/konnectivity-agent-bx24c" Nov 03 01:51:59.544255 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544180 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f15be6a7-766d-4eae-8e57-3a34850ca41b-config-volume\") pod \"dns-default-nbqt6\" (UID: \"f15be6a7-766d-4eae-8e57-3a34850ca41b\") " pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:59.544255 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544227 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-system-cni-dir\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.544255 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544243 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-run-ovn\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.544478 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544259 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7fh8\" (UniqueName: \"kubernetes.io/projected/1553379d-e8f5-4668-b524-13f5b93d0811-kube-api-access-s7fh8\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.544478 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544274 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a7b495c8-cb0d-4107-b99e-3f11833410d6-root\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.544478 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544308 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7b495c8-cb0d-4107-b99e-3f11833410d6-metrics-client-ca\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.544478 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544310 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3799f150-114a-4ed4-a955-e121465de3ab-konnectivity-ca\") pod \"konnectivity-agent-bx24c\" (UID: \"3799f150-114a-4ed4-a955-e121465de3ab\") " pod="kube-system/konnectivity-agent-bx24c" Nov 03 01:51:59.544478 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544370 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.544478 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544425 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.544478 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544449 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-slash\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.544478 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544473 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1553379d-e8f5-4668-b524-13f5b93d0811-env-overrides\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.544733 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544498 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c546c8d0-b721-45f9-b528-2e3490e84bef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:51:59.544733 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544613 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edf584ed-41f3-4fff-9e58-261b51e70b32-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rdhbb\" (UID: \"edf584ed-41f3-4fff-9e58-261b51e70b32\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:59.544733 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544629 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-etc-selinux\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:51:59.544733 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544645 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.544733 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544660 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-etc-openvswitch\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.544733 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544673 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd268007-d02d-44bc-8ad8-ad15135c1ae2-kubelet-config\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:59.544733 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544671 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1553379d-e8f5-4668-b524-13f5b93d0811-env-overrides\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.544733 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544691 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c546c8d0-b721-45f9-b528-2e3490e84bef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:51:59.544975 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544764 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-wtmp\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.544975 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544804 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-sys-fs\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:51:59.544975 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544854 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.544975 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544875 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srgxc\" (UniqueName: \"kubernetes.io/projected/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-api-access-srgxc\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:59.544975 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544891 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95977c23-899f-4892-8b57-c0e47ea6ec82-host\") pod \"node-ca-ggvl5\" (UID: \"95977c23-899f-4892-8b57-c0e47ea6ec82\") " pod="openshift-image-registry/node-ca-ggvl5" Nov 03 01:51:59.544975 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544908 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-cnibin\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.544975 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.544931 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-run-netns\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.545308 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.545095 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.545308 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.545139 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv668\" (UniqueName: \"kubernetes.io/projected/a134e24f-d1ad-467a-914e-0ff35ca0177b-kube-api-access-sv668\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.545308 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.545172 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729-host-slash\") pod \"iptables-alerter-47qlq\" (UID: \"ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729\") " pod="openshift-network-operator/iptables-alerter-47qlq" Nov 03 01:51:59.545308 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.545221 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-run-ovn-kubernetes\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.545308 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.545248 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c546c8d0-b721-45f9-b528-2e3490e84bef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:51:59.545308 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.545271 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-var-lib-cni-multus\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.545308 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.545304 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.545549 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.545334 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-textfile\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.545549 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.545395 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-textfile\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.545549 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.545426 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nnhh\" (UniqueName: \"kubernetes.io/projected/61b2893f-ece1-496b-b25d-56dc1fbb1f89-kube-api-access-9nnhh\") pod \"network-metrics-daemon-qmlx7\" (UID: \"61b2893f-ece1-496b-b25d-56dc1fbb1f89\") " pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:59.545549 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.545455 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c546c8d0-b721-45f9-b528-2e3490e84bef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:51:59.545549 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.545545 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:59.546002 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.545976 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzpf7\" (UniqueName: \"kubernetes.io/projected/a7b495c8-cb0d-4107-b99e-3f11833410d6-kube-api-access-lzpf7\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.546115 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546003 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c546c8d0-b721-45f9-b528-2e3490e84bef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:51:59.546115 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546030 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-sysctl-conf\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.546115 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546059 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-tuned\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.546334 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546222 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/edf584ed-41f3-4fff-9e58-261b51e70b32-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rdhbb\" (UID: \"edf584ed-41f3-4fff-9e58-261b51e70b32\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:59.546334 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546262 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:59.546334 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546291 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-tuned\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.546334 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546312 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729-iptables-alerter-script\") pod \"iptables-alerter-47qlq\" (UID: \"ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729\") " pod="openshift-network-operator/iptables-alerter-47qlq" Nov 03 01:51:59.546511 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546411 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.546511 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546453 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1553379d-e8f5-4668-b524-13f5b93d0811-ovnkube-script-lib\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.546511 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546486 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-sysctl-d\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.546649 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546515 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgm6s\" (UniqueName: \"kubernetes.io/projected/c546c8d0-b721-45f9-b528-2e3490e84bef-kube-api-access-mgm6s\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:51:59.546649 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546547 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-lib-modules\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.546649 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546554 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729-iptables-alerter-script\") pod \"iptables-alerter-47qlq\" (UID: \"ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729\") " pod="openshift-network-operator/iptables-alerter-47qlq" Nov 03 01:51:59.546649 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546580 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-run-k8s-cni-cncf-io\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.546649 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546614 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-multus-conf-dir\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.546863 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546658 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzprl\" (UniqueName: \"kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl\") pod \"network-check-target-rb2h2\" (UID: \"6ffde31b-d09a-4739-9bd4-abe007571006\") " pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:59.546863 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546712 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95977c23-899f-4892-8b57-c0e47ea6ec82-serviceca\") pod \"node-ca-ggvl5\" (UID: \"95977c23-899f-4892-8b57-c0e47ea6ec82\") " pod="openshift-image-registry/node-ca-ggvl5" Nov 03 01:51:59.546863 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546757 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14044133-e020-4562-84ec-9c09ce6e6e3c-cert\") pod \"ingress-canary-8sstb\" (UID: \"14044133-e020-4562-84ec-9c09ce6e6e3c\") " pod="openshift-ingress-canary/ingress-canary-8sstb" Nov 03 01:51:59.546863 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546789 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkbxw\" (UniqueName: \"kubernetes.io/projected/aafc0e0e-1526-4a94-a8c0-163e80562ea8-kube-api-access-pkbxw\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.546863 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546838 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a134e24f-d1ad-467a-914e-0ff35ca0177b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.547087 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546877 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-cni-netd\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.547087 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.546934 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.547757 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95977c23-899f-4892-8b57-c0e47ea6ec82-serviceca\") pod \"node-ca-ggvl5\" (UID: \"95977c23-899f-4892-8b57-c0e47ea6ec82\") " pod="openshift-image-registry/node-ca-ggvl5" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.547854 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7b495c8-cb0d-4107-b99e-3f11833410d6-sys\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.547893 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-accelerators-collector-config\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.547927 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-sys\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.547960 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.547996 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9clj\" (UniqueName: \"kubernetes.io/projected/0155a1d8-3fb6-48da-9319-c65514404dd1-kube-api-access-v9clj\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548107 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1553379d-e8f5-4668-b524-13f5b93d0811-ovn-node-metrics-cert\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548141 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15be6a7-766d-4eae-8e57-3a34850ca41b-tmp-dir\") pod \"dns-default-nbqt6\" (UID: \"f15be6a7-766d-4eae-8e57-3a34850ca41b\") " pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548172 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-modprobe-d\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548222 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548253 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-var-lib-cni-bin\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548279 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3799f150-114a-4ed4-a955-e121465de3ab-agent-certs\") pod \"konnectivity-agent-bx24c\" (UID: \"3799f150-114a-4ed4-a955-e121465de3ab\") " pod="kube-system/konnectivity-agent-bx24c" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548309 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548343 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548372 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-kubernetes\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548407 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-grpc-tls\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548438 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-kubelet\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548593 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c546c8d0-b721-45f9-b528-2e3490e84bef-system-cni-dir\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548621 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548654 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548688 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-multus-cni-dir\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548717 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-hostroot\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548742 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-registration-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548783 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-device-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548810 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd268007-d02d-44bc-8ad8-ad15135c1ae2-dbus\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548843 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edf584ed-41f3-4fff-9e58-261b51e70b32-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rdhbb\" (UID: \"edf584ed-41f3-4fff-9e58-261b51e70b32\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548871 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a134e24f-d1ad-467a-914e-0ff35ca0177b-config-out\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548902 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-var-lib-kubelet\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548933 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqfmb\" (UniqueName: \"kubernetes.io/projected/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729-kube-api-access-rqfmb\") pod \"iptables-alerter-47qlq\" (UID: \"ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729\") " pod="openshift-network-operator/iptables-alerter-47qlq" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548963 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-host\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.548994 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89-hosts-file\") pod \"node-resolver-skjrl\" (UID: \"be3a1b21-68d9-4a7b-b1e4-30669c2dbb89\") " pod="openshift-dns/node-resolver-skjrl" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.549020 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-socket-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.549054 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.549093 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-web-config\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.549127 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1553379d-e8f5-4668-b524-13f5b93d0811-ovnkube-config\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.549157 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.549183 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-tls\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.549286 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/307c837d-2477-4460-9606-46efe63be297-cni-binary-copy\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.549546 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-tls\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.549587 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.549699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.549618 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:59.551176 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.549752 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:59.551176 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.549758 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aafc0e0e-1526-4a94-a8c0-163e80562ea8-tmp\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.551176 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.549763 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15be6a7-766d-4eae-8e57-3a34850ca41b-tmp-dir\") pod \"dns-default-nbqt6\" (UID: \"f15be6a7-766d-4eae-8e57-3a34850ca41b\") " pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:59.551176 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.549926 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a134e24f-d1ad-467a-914e-0ff35ca0177b-config-out\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.551176 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.550099 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3799f150-114a-4ed4-a955-e121465de3ab-agent-certs\") pod \"konnectivity-agent-bx24c\" (UID: \"3799f150-114a-4ed4-a955-e121465de3ab\") " pod="kube-system/konnectivity-agent-bx24c" Nov 03 01:51:59.551176 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.550125 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd268007-d02d-44bc-8ad8-ad15135c1ae2-original-pull-secret\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:59.551176 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.550278 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1553379d-e8f5-4668-b524-13f5b93d0811-ovnkube-script-lib\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.551176 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.550474 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1553379d-e8f5-4668-b524-13f5b93d0811-ovnkube-config\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.551176 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.550552 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1553379d-e8f5-4668-b524-13f5b93d0811-ovn-node-metrics-cert\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.551176 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.550734 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/307c837d-2477-4460-9606-46efe63be297-cni-binary-copy\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.551176 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.550803 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-multus-socket-dir-parent\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.551176 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.550836 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aafc0e0e-1526-4a94-a8c0-163e80562ea8-tmp\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.551176 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.550962 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf67907b-cfa3-40ea-812d-4ea034671442-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:59.551176 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.551025 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:59.551562 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.551243 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf67907b-cfa3-40ea-812d-4ea034671442-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:59.584758 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.584731 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 01:51:59.584846 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.584793 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 01:51:59.584846 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.584824 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 01:51:59.584946 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.584877 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 01:51:59.584946 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.584892 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 01:51:59.585283 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.585262 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 01:51:59.585344 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.585288 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 01:51:59.585344 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.585301 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 01:51:59.585344 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.585305 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 01:51:59.588661 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.588641 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 01:51:59.588797 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.588782 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 01:51:59.588864 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.588852 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 01:51:59.588904 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.588877 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 01:51:59.591042 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.591022 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-accelerators-collector-config\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.591100 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.591046 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.591100 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.591068 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.591100 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.591082 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.591100 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.591048 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.591239 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.591097 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:59.592111 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.592091 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.592145 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.592112 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:59.592313 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.592292 5463 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:51:59.592470 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:51:59.592405 5463 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:59.593054 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.593041 5463 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:51:59.593101 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:51:59.593071 5463 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-69.us-west-2.compute.internal" Nov 03 01:51:59.594240 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.594225 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fad00740-ad1a-4623-ba88-2b2319c14266-metrics-client-ca\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:59.594372 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.594360 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f15be6a7-766d-4eae-8e57-3a34850ca41b-config-volume\") pod \"dns-default-nbqt6\" (UID: \"f15be6a7-766d-4eae-8e57-3a34850ca41b\") " pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:59.594515 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.594502 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7b495c8-cb0d-4107-b99e-3f11833410d6-metrics-client-ca\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.595325 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.595310 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edf584ed-41f3-4fff-9e58-261b51e70b32-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rdhbb\" (UID: \"edf584ed-41f3-4fff-9e58-261b51e70b32\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:59.596561 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.596545 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c7dg\" (UniqueName: \"kubernetes.io/projected/95977c23-899f-4892-8b57-c0e47ea6ec82-kube-api-access-8c7dg\") pod \"node-ca-ggvl5\" (UID: \"95977c23-899f-4892-8b57-c0e47ea6ec82\") " pod="openshift-image-registry/node-ca-ggvl5" Nov 03 01:51:59.597029 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.597012 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q7cl\" (UniqueName: \"kubernetes.io/projected/edf584ed-41f3-4fff-9e58-261b51e70b32-kube-api-access-4q7cl\") pod \"openshift-state-metrics-7bd7489957-rdhbb\" (UID: \"edf584ed-41f3-4fff-9e58-261b51e70b32\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:59.597118 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.597103 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xltsq\" (UniqueName: \"kubernetes.io/projected/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89-kube-api-access-xltsq\") pod \"node-resolver-skjrl\" (UID: \"be3a1b21-68d9-4a7b-b1e4-30669c2dbb89\") " pod="openshift-dns/node-resolver-skjrl" Nov 03 01:51:59.597264 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.597244 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnj6b\" (UniqueName: \"kubernetes.io/projected/cf67907b-cfa3-40ea-812d-4ea034671442-kube-api-access-xnj6b\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:59.598201 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.598170 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97v2w\" (UniqueName: \"kubernetes.io/projected/fcd16c62-1c32-448a-9f84-45fb913f26ba-kube-api-access-97v2w\") pod \"monitoring-plugin-77f4f76594-98ckh\" (UID: \"fcd16c62-1c32-448a-9f84-45fb913f26ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-98ckh" Nov 03 01:51:59.598496 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.598476 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.599100 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.599083 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.602530 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.602512 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52pmv\" (UniqueName: \"kubernetes.io/projected/f15be6a7-766d-4eae-8e57-3a34850ca41b-kube-api-access-52pmv\") pod \"dns-default-nbqt6\" (UID: \"f15be6a7-766d-4eae-8e57-3a34850ca41b\") " pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:59.605867 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.605846 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgm6s\" (UniqueName: \"kubernetes.io/projected/c546c8d0-b721-45f9-b528-2e3490e84bef-kube-api-access-mgm6s\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:51:59.606429 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.606409 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv668\" (UniqueName: \"kubernetes.io/projected/a134e24f-d1ad-467a-914e-0ff35ca0177b-kube-api-access-sv668\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.606702 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.606688 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzpf7\" (UniqueName: \"kubernetes.io/projected/a7b495c8-cb0d-4107-b99e-3f11833410d6-kube-api-access-lzpf7\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.607312 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.607297 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqpwg\" (UniqueName: \"kubernetes.io/projected/14044133-e020-4562-84ec-9c09ce6e6e3c-kube-api-access-nqpwg\") pod \"ingress-canary-8sstb\" (UID: \"14044133-e020-4562-84ec-9c09ce6e6e3c\") " pod="openshift-ingress-canary/ingress-canary-8sstb" Nov 03 01:51:59.607488 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.607474 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7fh8\" (UniqueName: \"kubernetes.io/projected/1553379d-e8f5-4668-b524-13f5b93d0811-kube-api-access-s7fh8\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.608148 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.608133 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mbln\" (UniqueName: \"kubernetes.io/projected/fad00740-ad1a-4623-ba88-2b2319c14266-kube-api-access-2mbln\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:59.608322 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.608308 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nnhh\" (UniqueName: \"kubernetes.io/projected/61b2893f-ece1-496b-b25d-56dc1fbb1f89-kube-api-access-9nnhh\") pod \"network-metrics-daemon-qmlx7\" (UID: \"61b2893f-ece1-496b-b25d-56dc1fbb1f89\") " pod="openshift-multus/network-metrics-daemon-qmlx7" Nov 03 01:51:59.609469 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.609439 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srgxc\" (UniqueName: \"kubernetes.io/projected/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-api-access-srgxc\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:59.609547 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.609485 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k759x\" (UniqueName: \"kubernetes.io/projected/307c837d-2477-4460-9606-46efe63be297-kube-api-access-k759x\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.609717 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.609698 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkbxw\" (UniqueName: \"kubernetes.io/projected/aafc0e0e-1526-4a94-a8c0-163e80562ea8-kube-api-access-pkbxw\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.609803 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.609783 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqfmb\" (UniqueName: \"kubernetes.io/projected/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729-kube-api-access-rqfmb\") pod \"iptables-alerter-47qlq\" (UID: \"ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729\") " pod="openshift-network-operator/iptables-alerter-47qlq" Nov 03 01:51:59.610256 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.610239 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzprl\" (UniqueName: \"kubernetes.io/projected/6ffde31b-d09a-4739-9bd4-abe007571006-kube-api-access-gzprl\") pod \"network-check-target-rb2h2\" (UID: \"6ffde31b-d09a-4739-9bd4-abe007571006\") " pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:51:59.610362 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.610349 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9clj\" (UniqueName: \"kubernetes.io/projected/0155a1d8-3fb6-48da-9319-c65514404dd1-kube-api-access-v9clj\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:51:59.635993 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.635935 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 01:51:59.635993 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.635954 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 01:51:59.635993 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.635975 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 01:51:59.636146 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.635996 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 01:51:59.636146 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636051 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gq552\"" Nov 03 01:51:59.636146 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636060 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 01:51:59.636395 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636281 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 01:51:59.636395 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636313 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 01:51:59.636395 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636336 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 01:51:59.636395 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636346 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 01:51:59.636395 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636336 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 01:51:59.636395 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636366 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 01:51:59.636657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636584 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 01:51:59.636657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636604 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 01:51:59.636657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636619 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 01:51:59.636657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636610 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8rsahm0n9sh3l\"" Nov 03 01:51:59.636657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636648 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 01:51:59.636870 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636726 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 01:51:59.636870 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636726 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 01:51:59.636870 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636781 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nj9dq\"" Nov 03 01:51:59.636870 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636787 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 01:51:59.636870 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636788 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-d63c6q864pbr4\"" Nov 03 01:51:59.636870 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636823 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zjrsl\"" Nov 03 01:51:59.636870 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636832 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-srm4x\"" Nov 03 01:51:59.636870 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636825 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8rp9w\"" Nov 03 01:51:59.636870 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636827 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 01:51:59.636870 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636854 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bsx6s\"" Nov 03 01:51:59.636870 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636833 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 01:51:59.636870 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636822 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-v5ks7\"" Nov 03 01:51:59.637248 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636863 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 01:51:59.637248 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636909 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-m7jn6\"" Nov 03 01:51:59.637248 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.636959 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 01:51:59.637248 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.637011 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 01:51:59.637826 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.637812 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a134e24f-d1ad-467a-914e-0ff35ca0177b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.641060 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.640984 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.641060 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.641002 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-web-config\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.641060 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.641034 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edf584ed-41f3-4fff-9e58-261b51e70b32-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rdhbb\" (UID: \"edf584ed-41f3-4fff-9e58-261b51e70b32\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:59.641238 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.641093 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.641238 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.641108 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-tls\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.641238 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.641127 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:59.641238 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.641128 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-tls\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:59.641238 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.641127 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14044133-e020-4562-84ec-9c09ce6e6e3c-cert\") pod \"ingress-canary-8sstb\" (UID: \"14044133-e020-4562-84ec-9c09ce6e6e3c\") " pod="openshift-ingress-canary/ingress-canary-8sstb" Nov 03 01:51:59.641238 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.641128 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:59.641455 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.641283 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-grpc-tls\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:59.641455 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.641287 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:59.642478 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.642464 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f15be6a7-766d-4eae-8e57-3a34850ca41b-metrics-tls\") pod \"dns-default-nbqt6\" (UID: \"f15be6a7-766d-4eae-8e57-3a34850ca41b\") " pod="openshift-dns/dns-default-nbqt6" Nov 03 01:51:59.642518 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.642476 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.642554 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.642525 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.643690 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.643670 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.643690 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.643681 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fcd16c62-1c32-448a-9f84-45fb913f26ba-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-98ckh\" (UID: \"fcd16c62-1c32-448a-9f84-45fb913f26ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-98ckh" Nov 03 01:51:59.643822 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.643807 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-config\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.644699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.644682 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.644699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.644691 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:59.644903 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.644891 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.645540 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.645527 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:51:59.647102 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.647087 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37dd5ea7-75ae-4414-8c9a-90fe106cfd48-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hr8gq\" (UID: \"37dd5ea7-75ae-4414-8c9a-90fe106cfd48\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hr8gq" Nov 03 01:51:59.647224 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.647205 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fad00740-ad1a-4623-ba88-2b2319c14266-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54564794f6-z44tk\" (UID: \"fad00740-ad1a-4623-ba88-2b2319c14266\") " pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:51:59.647264 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.647230 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/edf584ed-41f3-4fff-9e58-261b51e70b32-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rdhbb\" (UID: \"edf584ed-41f3-4fff-9e58-261b51e70b32\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rdhbb" Nov 03 01:51:59.651390 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651371 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-multus-cni-dir\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.651443 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651395 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-hostroot\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.651443 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651418 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-registration-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:51:59.651443 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651433 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-device-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:51:59.651579 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651455 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd268007-d02d-44bc-8ad8-ad15135c1ae2-dbus\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:59.651579 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651479 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-var-lib-kubelet\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.651579 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651480 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-hostroot\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.651579 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651479 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-multus-cni-dir\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.651579 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651504 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-host\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.651579 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651514 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-var-lib-kubelet\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.651579 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651502 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-registration-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:51:59.651579 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651529 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd268007-d02d-44bc-8ad8-ad15135c1ae2-dbus\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:59.651579 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651524 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-device-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:51:59.651579 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651526 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89-hosts-file\") pod \"node-resolver-skjrl\" (UID: \"be3a1b21-68d9-4a7b-b1e4-30669c2dbb89\") " pod="openshift-dns/node-resolver-skjrl" Nov 03 01:51:59.651579 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651551 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-host\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.651579 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651569 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be3a1b21-68d9-4a7b-b1e4-30669c2dbb89-hosts-file\") pod \"node-resolver-skjrl\" (UID: \"be3a1b21-68d9-4a7b-b1e4-30669c2dbb89\") " pod="openshift-dns/node-resolver-skjrl" Nov 03 01:51:59.651579 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651583 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-socket-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651605 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-multus-socket-dir-parent\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651624 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-run-multus-certs\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651640 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-etc-kubernetes\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651655 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-systemd\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651676 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-run-multus-certs\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651681 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-multus-socket-dir-parent\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651694 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-systemd\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651696 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-etc-kubernetes\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651683 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-run\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651714 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-run\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651734 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-socket-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651739 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cf67907b-cfa3-40ea-812d-4ea034671442-crio-socket\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651761 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651777 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-run-systemd\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651790 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-var-lib-openvswitch\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651793 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cf67907b-cfa3-40ea-812d-4ea034671442-crio-socket\") pod \"insights-runtime-extractor-mdxfm\" (UID: \"cf67907b-cfa3-40ea-812d-4ea034671442\") " pod="openshift-insights/insights-runtime-extractor-mdxfm" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651803 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-node-log\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651827 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-node-log\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651830 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-log-socket\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651836 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-run-systemd\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651833 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651854 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-log-socket\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651858 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-cni-bin\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651853 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-var-lib-openvswitch\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651877 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c546c8d0-b721-45f9-b528-2e3490e84bef-cnibin\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651894 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-systemd-units\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651908 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-run-openvswitch\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.651916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651926 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-cni-bin\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651928 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-var-lib-kubelet\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651927 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c546c8d0-b721-45f9-b528-2e3490e84bef-cnibin\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651951 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-run-openvswitch\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651955 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-run-netns\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651979 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-sysconfig\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651952 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-systemd-units\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.651980 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-run-netns\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652002 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-system-cni-dir\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652027 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-var-lib-kubelet\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652039 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-sysconfig\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652037 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-system-cni-dir\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652060 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-run-ovn\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652078 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a7b495c8-cb0d-4107-b99e-3f11833410d6-root\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652093 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-slash\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652109 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-etc-selinux\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652117 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-run-ovn\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652127 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-etc-openvswitch\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652144 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a7b495c8-cb0d-4107-b99e-3f11833410d6-root\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652154 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-slash\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652157 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-etc-openvswitch\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652166 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-etc-selinux\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652156 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd268007-d02d-44bc-8ad8-ad15135c1ae2-kubelet-config\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652227 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd268007-d02d-44bc-8ad8-ad15135c1ae2-kubelet-config\") pod \"global-pull-secret-syncer-r2z98\" (UID: \"cd268007-d02d-44bc-8ad8-ad15135c1ae2\") " pod="kube-system/global-pull-secret-syncer-r2z98" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652228 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-wtmp\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652251 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-sys-fs\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652260 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a7b495c8-cb0d-4107-b99e-3f11833410d6-node-exporter-wtmp\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652266 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95977c23-899f-4892-8b57-c0e47ea6ec82-host\") pod \"node-ca-ggvl5\" (UID: \"95977c23-899f-4892-8b57-c0e47ea6ec82\") " pod="openshift-image-registry/node-ca-ggvl5" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652288 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-cnibin\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652305 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0155a1d8-3fb6-48da-9319-c65514404dd1-sys-fs\") pod \"aws-ebs-csi-driver-node-4vbxc\" (UID: \"0155a1d8-3fb6-48da-9319-c65514404dd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vbxc" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652309 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95977c23-899f-4892-8b57-c0e47ea6ec82-host\") pod \"node-ca-ggvl5\" (UID: \"95977c23-899f-4892-8b57-c0e47ea6ec82\") " pod="openshift-image-registry/node-ca-ggvl5" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652306 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-run-netns\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652323 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-run-netns\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652345 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-cnibin\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652351 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729-host-slash\") pod \"iptables-alerter-47qlq\" (UID: \"ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729\") " pod="openshift-network-operator/iptables-alerter-47qlq" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652368 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729-host-slash\") pod \"iptables-alerter-47qlq\" (UID: \"ff4b2b6f-bac7-4e47-bce0-7c7ab61c8729\") " pod="openshift-network-operator/iptables-alerter-47qlq" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652367 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-run-ovn-kubernetes\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652387 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-run-ovn-kubernetes\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652387 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c546c8d0-b721-45f9-b528-2e3490e84bef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652411 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-var-lib-cni-multus\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652413 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c546c8d0-b721-45f9-b528-2e3490e84bef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652437 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-var-lib-cni-multus\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652455 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652472 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-sysctl-conf\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652524 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-sysctl-d\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652542 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-lib-modules\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652564 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-run-k8s-cni-cncf-io\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652574 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652585 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-multus-conf-dir\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652607 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-cni-netd\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652614 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-run-k8s-cni-cncf-io\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652626 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7b495c8-cb0d-4107-b99e-3f11833410d6-sys\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652647 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-sys\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652650 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-multus-conf-dir\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652664 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-lib-modules\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652672 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-modprobe-d\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652675 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-sysctl-conf\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652690 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7b495c8-cb0d-4107-b99e-3f11833410d6-sys\") pod \"node-exporter-w7h88\" (UID: \"a7b495c8-cb0d-4107-b99e-3f11833410d6\") " pod="openshift-monitoring/node-exporter-w7h88" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652667 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-sysctl-d\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652702 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-var-lib-cni-bin\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652697 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-cni-netd\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.654570 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652709 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-sys\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.654570 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652728 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-kubernetes\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.654570 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652735 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/307c837d-2477-4460-9606-46efe63be297-host-var-lib-cni-bin\") pod \"multus-g8nt7\" (UID: \"307c837d-2477-4460-9606-46efe63be297\") " pod="openshift-multus/multus-g8nt7" Nov 03 01:51:59.654570 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652745 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-modprobe-d\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.654570 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652754 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-kubelet\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.654570 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652775 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c546c8d0-b721-45f9-b528-2e3490e84bef-system-cni-dir\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:51:59.654570 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652782 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aafc0e0e-1526-4a94-a8c0-163e80562ea8-etc-kubernetes\") pod \"tuned-jvq7p\" (UID: \"aafc0e0e-1526-4a94-a8c0-163e80562ea8\") " pod="openshift-cluster-node-tuning-operator/tuned-jvq7p" Nov 03 01:51:59.654570 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652786 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1553379d-e8f5-4668-b524-13f5b93d0811-host-kubelet\") pod \"ovnkube-node-mbmwx\" (UID: \"1553379d-e8f5-4668-b524-13f5b93d0811\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:51:59.654570 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.652827 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c546c8d0-b721-45f9-b528-2e3490e84bef-system-cni-dir\") pod \"multus-additional-cni-plugins-c6nzj\" (UID: \"c546c8d0-b721-45f9-b528-2e3490e84bef\") " pod="openshift-multus/multus-additional-cni-plugins-c6nzj" Nov 03 01:51:59.654570 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.653666 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:59.795150 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.795134 5463 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:51:59.807645 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.807624 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:59.821505 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:51:59.821487 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nbqt6" Nov 03 01:52:00.080545 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:00.080513 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:52:00.137832 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:00.137750 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:00.286118 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:00.286093 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:01.969316 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:01.969280 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-98ckh" Nov 03 01:52:01.974349 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:01.974324 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-98ckh" Nov 03 01:52:02.277428 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:02.277354 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:52:02.573043 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:02.572941 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:02.724141 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:02.724110 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:02.866627 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:02.866536 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:03.052269 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:03.052233 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:52:03.058568 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:03.058543 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-54564794f6-z44tk" Nov 03 01:52:03.436723 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:03.436696 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:52:03.436919 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:03.436907 5463 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 01:52:03.520080 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:03.520052 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a134e24f-d1ad-467a-914e-0ff35ca0177b","Type":"ContainerStarted","Data":"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b"} Nov 03 01:52:03.587592 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:03.587557 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:03.784762 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:03.784725 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:04.526818 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:04.526785 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a134e24f-d1ad-467a-914e-0ff35ca0177b","Type":"ContainerStarted","Data":"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031"} Nov 03 01:52:04.527313 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:04.526856 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a134e24f-d1ad-467a-914e-0ff35ca0177b","Type":"ContainerStarted","Data":"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea"} Nov 03 01:52:04.527313 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:04.526897 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a134e24f-d1ad-467a-914e-0ff35ca0177b","Type":"ContainerStarted","Data":"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6"} Nov 03 01:52:04.527313 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:04.526947 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a134e24f-d1ad-467a-914e-0ff35ca0177b","Type":"ContainerStarted","Data":"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447"} Nov 03 01:52:04.527313 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:04.526987 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a134e24f-d1ad-467a-914e-0ff35ca0177b","Type":"ContainerStarted","Data":"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965"} Nov 03 01:52:04.537319 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:04.537291 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:04.681921 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:04.681872 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:04.782064 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:04.781993 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:09.282992 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:09.282893 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:11.738344 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:11.738310 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:11.885386 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:11.885351 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:12.184401 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:12.184369 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:12.239105 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:12.239082 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:12.572796 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:12.572733 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:13.005166 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:52:13.005131 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 01:52:14.061896 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:14.061862 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:14.212315 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:14.212288 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:14.662866 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:14.662835 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:14.820990 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:14.820957 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:14.968801 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:14.968724 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:17.492041 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:17.492005 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:17.642964 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:17.642923 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:18.702467 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:18.702431 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:18.845403 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:18.845367 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:19.273897 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:19.273868 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:19.413069 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:19.413038 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:23.116632 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:23.116597 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:23.144054 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:23.144032 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bx24c" Nov 03 01:52:23.271113 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:23.271081 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:23.734866 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:23.734841 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbmwx" Nov 03 01:52:27.117683 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:27.117649 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:27.262426 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:27.262395 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:27.893017 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:27.892984 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 01:52:27.893386 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:27.893348 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="prometheus" containerID="cri-o://03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b" gracePeriod=600 Nov 03 01:52:27.893458 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:27.893394 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="kube-rbac-proxy" containerID="cri-o://de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea" gracePeriod=600 Nov 03 01:52:27.893458 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:27.893431 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="thanos-sidecar" containerID="cri-o://8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447" gracePeriod=600 Nov 03 01:52:27.893552 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:27.893457 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="kube-rbac-proxy-web" containerID="cri-o://be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6" gracePeriod=600 Nov 03 01:52:27.893552 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:27.893465 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031" gracePeriod=600 Nov 03 01:52:27.893552 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:27.893433 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="config-reloader" containerID="cri-o://2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965" gracePeriod=600 Nov 03 01:52:28.143564 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.143542 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.170618 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.170586 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a134e24f-d1ad-467a-914e-0ff35ca0177b-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a134e24f-d1ad-467a-914e-0ff35ca0177b-config-out\") pod \"a134e24f-d1ad-467a-914e-0ff35ca0177b\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " Nov 03 01:52:28.170724 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.170631 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-web-config\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-web-config\") pod \"a134e24f-d1ad-467a-914e-0ff35ca0177b\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " Nov 03 01:52:28.170724 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.170672 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-2\") pod \"a134e24f-d1ad-467a-914e-0ff35ca0177b\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " Nov 03 01:52:28.170724 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.170707 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-grpc-tls\") pod \"a134e24f-d1ad-467a-914e-0ff35ca0177b\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " Nov 03 01:52:28.170860 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.170738 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-tls\") pod \"a134e24f-d1ad-467a-914e-0ff35ca0177b\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " Nov 03 01:52:28.170954 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.170880 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-kube-rbac-proxy\") pod \"a134e24f-d1ad-467a-914e-0ff35ca0177b\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " Nov 03 01:52:28.171072 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.171001 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-serving-certs-ca-bundle\") pod \"a134e24f-d1ad-467a-914e-0ff35ca0177b\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " Nov 03 01:52:28.171072 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.171020 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-2" pod "a134e24f-d1ad-467a-914e-0ff35ca0177b" (UID: "a134e24f-d1ad-467a-914e-0ff35ca0177b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:52:28.171722 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.171339 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-kubelet-serving-ca-bundle\") pod \"a134e24f-d1ad-467a-914e-0ff35ca0177b\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " Nov 03 01:52:28.171906 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.171886 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a134e24f-d1ad-467a-914e-0ff35ca0177b-tls-assets\" (UniqueName: \"kubernetes.io/projected/a134e24f-d1ad-467a-914e-0ff35ca0177b-tls-assets\") pod \"a134e24f-d1ad-467a-914e-0ff35ca0177b\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " Nov 03 01:52:28.172387 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.172046 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a134e24f-d1ad-467a-914e-0ff35ca0177b\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " Nov 03 01:52:28.172556 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.172537 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a134e24f-d1ad-467a-914e-0ff35ca0177b\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " Nov 03 01:52:28.172669 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.172653 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-metrics-client-ca\") pod \"a134e24f-d1ad-467a-914e-0ff35ca0177b\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " Nov 03 01:52:28.172794 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.172777 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-metrics-client-certs\") pod \"a134e24f-d1ad-467a-914e-0ff35ca0177b\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " Nov 03 01:52:28.172914 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.172898 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-thanos-prometheus-http-client-file\") pod \"a134e24f-d1ad-467a-914e-0ff35ca0177b\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " Nov 03 01:52:28.173386 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.173366 5463 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:28.173670 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.171460 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-serving-certs-ca-bundle" pod "a134e24f-d1ad-467a-914e-0ff35ca0177b" (UID: "a134e24f-d1ad-467a-914e-0ff35ca0177b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:52:28.173756 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.171699 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-kubelet-serving-ca-bundle" pod "a134e24f-d1ad-467a-914e-0ff35ca0177b" (UID: "a134e24f-d1ad-467a-914e-0ff35ca0177b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:52:28.173756 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.173638 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-metrics-client-ca" pod "a134e24f-d1ad-467a-914e-0ff35ca0177b" (UID: "a134e24f-d1ad-467a-914e-0ff35ca0177b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:52:28.173756 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.173725 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-grpc-tls" pod "a134e24f-d1ad-467a-914e-0ff35ca0177b" (UID: "a134e24f-d1ad-467a-914e-0ff35ca0177b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:28.173928 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.173805 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-tls" pod "a134e24f-d1ad-467a-914e-0ff35ca0177b" (UID: "a134e24f-d1ad-467a-914e-0ff35ca0177b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:28.173928 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.173854 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a134e24f-d1ad-467a-914e-0ff35ca0177b-config-out" pod "a134e24f-d1ad-467a-914e-0ff35ca0177b" (UID: "a134e24f-d1ad-467a-914e-0ff35ca0177b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:52:28.174046 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.174024 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-kube-rbac-proxy" pod "a134e24f-d1ad-467a-914e-0ff35ca0177b" (UID: "a134e24f-d1ad-467a-914e-0ff35ca0177b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:28.175923 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.175892 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a134e24f-d1ad-467a-914e-0ff35ca0177b-tls-assets" pod "a134e24f-d1ad-467a-914e-0ff35ca0177b" (UID: "a134e24f-d1ad-467a-914e-0ff35ca0177b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:52:28.175923 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.175918 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-kube-rbac-proxy-web" pod "a134e24f-d1ad-467a-914e-0ff35ca0177b" (UID: "a134e24f-d1ad-467a-914e-0ff35ca0177b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:28.176095 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.176008 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-thanos-sidecar-tls" pod "a134e24f-d1ad-467a-914e-0ff35ca0177b" (UID: "a134e24f-d1ad-467a-914e-0ff35ca0177b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:28.176095 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.176079 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-metrics-client-certs" pod "a134e24f-d1ad-467a-914e-0ff35ca0177b" (UID: "a134e24f-d1ad-467a-914e-0ff35ca0177b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:28.176419 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.176395 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-thanos-prometheus-http-client-file" pod "a134e24f-d1ad-467a-914e-0ff35ca0177b" (UID: "a134e24f-d1ad-467a-914e-0ff35ca0177b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:28.183396 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.183372 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-web-config" pod "a134e24f-d1ad-467a-914e-0ff35ca0177b" (UID: "a134e24f-d1ad-467a-914e-0ff35ca0177b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:28.217819 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.217785 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:28.273977 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.273949 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-0\") pod \"a134e24f-d1ad-467a-914e-0ff35ca0177b\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " Nov 03 01:52:28.274086 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.273997 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-1\") pod \"a134e24f-d1ad-467a-914e-0ff35ca0177b\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " Nov 03 01:52:28.274086 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.274029 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a134e24f-d1ad-467a-914e-0ff35ca0177b-kube-api-access-sv668\" (UniqueName: \"kubernetes.io/projected/a134e24f-d1ad-467a-914e-0ff35ca0177b-kube-api-access-sv668\") pod \"a134e24f-d1ad-467a-914e-0ff35ca0177b\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " Nov 03 01:52:28.274086 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.274060 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-trusted-ca-bundle\") pod \"a134e24f-d1ad-467a-914e-0ff35ca0177b\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " Nov 03 01:52:28.274275 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.274094 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-config\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-config\") pod \"a134e24f-d1ad-467a-914e-0ff35ca0177b\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " Nov 03 01:52:28.274275 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.274124 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-db\") pod \"a134e24f-d1ad-467a-914e-0ff35ca0177b\" (UID: \"a134e24f-d1ad-467a-914e-0ff35ca0177b\") " Nov 03 01:52:28.274745 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.274376 5463 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a134e24f-d1ad-467a-914e-0ff35ca0177b-config-out\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:28.274745 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.274396 5463 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-web-config\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:28.274745 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.274408 5463 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-grpc-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:28.274745 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.274418 5463 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:28.274745 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.274429 5463 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-kube-rbac-proxy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:28.274745 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.274442 5463 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:28.274745 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.274457 5463 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:28.274745 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.274472 5463 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a134e24f-d1ad-467a-914e-0ff35ca0177b-tls-assets\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:28.274745 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.274486 5463 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:28.274745 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.274500 5463 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:28.274745 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.274514 5463 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-configmap-metrics-client-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:28.274745 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.274529 5463 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-secret-metrics-client-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:28.274745 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.274544 5463 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:28.275493 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.274902 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-1" pod "a134e24f-d1ad-467a-914e-0ff35ca0177b" (UID: "a134e24f-d1ad-467a-914e-0ff35ca0177b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:52:28.275493 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.274682 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-trusted-ca-bundle" pod "a134e24f-d1ad-467a-914e-0ff35ca0177b" (UID: "a134e24f-d1ad-467a-914e-0ff35ca0177b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:52:28.275797 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.275767 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-0" pod "a134e24f-d1ad-467a-914e-0ff35ca0177b" (UID: "a134e24f-d1ad-467a-914e-0ff35ca0177b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:52:28.276743 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.276718 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-db" pod "a134e24f-d1ad-467a-914e-0ff35ca0177b" (UID: "a134e24f-d1ad-467a-914e-0ff35ca0177b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:52:28.276838 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.276759 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a134e24f-d1ad-467a-914e-0ff35ca0177b-kube-api-access-sv668" pod "a134e24f-d1ad-467a-914e-0ff35ca0177b" (UID: "a134e24f-d1ad-467a-914e-0ff35ca0177b"). InnerVolumeSpecName "kube-api-access-sv668". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:52:28.276838 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.276807 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-config" pod "a134e24f-d1ad-467a-914e-0ff35ca0177b" (UID: "a134e24f-d1ad-467a-914e-0ff35ca0177b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:52:28.368469 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.368433 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:28.375595 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.375568 5463 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:28.375595 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.375593 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv668\" (UniqueName: \"kubernetes.io/projected/a134e24f-d1ad-467a-914e-0ff35ca0177b-kube-api-access-sv668\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:28.375749 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.375610 5463 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:28.375749 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.375624 5463 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a134e24f-d1ad-467a-914e-0ff35ca0177b-config\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:28.375749 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.375637 5463 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-db\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:28.375749 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.375650 5463 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a134e24f-d1ad-467a-914e-0ff35ca0177b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:52:28.612851 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.612822 5463 generic.go:356] "Generic (PLEG): container finished" podID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerID="ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031" exitCode=0 Nov 03 01:52:28.612851 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.612847 5463 generic.go:356] "Generic (PLEG): container finished" podID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerID="de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea" exitCode=0 Nov 03 01:52:28.612851 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.612853 5463 generic.go:356] "Generic (PLEG): container finished" podID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerID="be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6" exitCode=0 Nov 03 01:52:28.612851 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.612859 5463 generic.go:356] "Generic (PLEG): container finished" podID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerID="8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447" exitCode=0 Nov 03 01:52:28.613135 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.612865 5463 generic.go:356] "Generic (PLEG): container finished" podID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerID="2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965" exitCode=0 Nov 03 01:52:28.613135 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.612875 5463 generic.go:356] "Generic (PLEG): container finished" podID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerID="03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b" exitCode=0 Nov 03 01:52:28.613135 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.612900 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a134e24f-d1ad-467a-914e-0ff35ca0177b","Type":"ContainerDied","Data":"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031"} Nov 03 01:52:28.613135 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.612912 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.613135 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.613007 5463 scope.go:122] "RemoveContainer" containerID="ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031" Nov 03 01:52:28.613135 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.612987 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a134e24f-d1ad-467a-914e-0ff35ca0177b","Type":"ContainerDied","Data":"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea"} Nov 03 01:52:28.613416 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.613151 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a134e24f-d1ad-467a-914e-0ff35ca0177b","Type":"ContainerDied","Data":"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6"} Nov 03 01:52:28.613416 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.613217 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a134e24f-d1ad-467a-914e-0ff35ca0177b","Type":"ContainerDied","Data":"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447"} Nov 03 01:52:28.613416 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.613245 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a134e24f-d1ad-467a-914e-0ff35ca0177b","Type":"ContainerDied","Data":"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965"} Nov 03 01:52:28.613416 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.613273 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a134e24f-d1ad-467a-914e-0ff35ca0177b","Type":"ContainerDied","Data":"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b"} Nov 03 01:52:28.613416 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.613307 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a134e24f-d1ad-467a-914e-0ff35ca0177b","Type":"ContainerDied","Data":"950008f8e686ec9f369a11655f4dd77308b491c84e4912ddded048cc3bfbccee"} Nov 03 01:52:28.621012 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.620997 5463 scope.go:122] "RemoveContainer" containerID="de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea" Nov 03 01:52:28.627462 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.627443 5463 scope.go:122] "RemoveContainer" containerID="be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6" Nov 03 01:52:28.633452 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.633437 5463 scope.go:122] "RemoveContainer" containerID="8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447" Nov 03 01:52:28.639427 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.639399 5463 scope.go:122] "RemoveContainer" containerID="2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965" Nov 03 01:52:28.645560 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.645545 5463 scope.go:122] "RemoveContainer" containerID="03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b" Nov 03 01:52:28.652416 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.652401 5463 scope.go:122] "RemoveContainer" containerID="e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066" Nov 03 01:52:28.658401 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.658385 5463 scope.go:122] "RemoveContainer" containerID="ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031" Nov 03 01:52:28.658659 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:52:28.658639 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031\": container with ID starting with ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031 not found: ID does not exist" containerID="ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031" Nov 03 01:52:28.658735 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.658669 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031"} err="failed to get container status \"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031\": rpc error: code = NotFound desc = could not find container \"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031\": container with ID starting with ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031 not found: ID does not exist" Nov 03 01:52:28.658735 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.658711 5463 scope.go:122] "RemoveContainer" containerID="de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea" Nov 03 01:52:28.658991 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:52:28.658974 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea\": container with ID starting with de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea not found: ID does not exist" containerID="de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea" Nov 03 01:52:28.659029 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.658999 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea"} err="failed to get container status \"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea\": rpc error: code = NotFound desc = could not find container \"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea\": container with ID starting with de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea not found: ID does not exist" Nov 03 01:52:28.659029 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.659014 5463 scope.go:122] "RemoveContainer" containerID="be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6" Nov 03 01:52:28.659274 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:52:28.659257 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6\": container with ID starting with be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6 not found: ID does not exist" containerID="be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6" Nov 03 01:52:28.659332 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.659279 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6"} err="failed to get container status \"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6\": rpc error: code = NotFound desc = could not find container \"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6\": container with ID starting with be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6 not found: ID does not exist" Nov 03 01:52:28.659332 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.659294 5463 scope.go:122] "RemoveContainer" containerID="8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447" Nov 03 01:52:28.659519 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:52:28.659503 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447\": container with ID starting with 8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447 not found: ID does not exist" containerID="8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447" Nov 03 01:52:28.659581 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.659527 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447"} err="failed to get container status \"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447\": rpc error: code = NotFound desc = could not find container \"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447\": container with ID starting with 8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447 not found: ID does not exist" Nov 03 01:52:28.659581 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.659549 5463 scope.go:122] "RemoveContainer" containerID="2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965" Nov 03 01:52:28.659770 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:52:28.659753 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965\": container with ID starting with 2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965 not found: ID does not exist" containerID="2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965" Nov 03 01:52:28.659807 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.659774 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965"} err="failed to get container status \"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965\": rpc error: code = NotFound desc = could not find container \"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965\": container with ID starting with 2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965 not found: ID does not exist" Nov 03 01:52:28.659807 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.659787 5463 scope.go:122] "RemoveContainer" containerID="03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b" Nov 03 01:52:28.659989 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:52:28.659972 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b\": container with ID starting with 03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b not found: ID does not exist" containerID="03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b" Nov 03 01:52:28.660044 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.659995 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b"} err="failed to get container status \"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b\": rpc error: code = NotFound desc = could not find container \"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b\": container with ID starting with 03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b not found: ID does not exist" Nov 03 01:52:28.660044 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.660013 5463 scope.go:122] "RemoveContainer" containerID="e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066" Nov 03 01:52:28.660232 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:52:28.660217 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066\": container with ID starting with e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066 not found: ID does not exist" containerID="e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066" Nov 03 01:52:28.660268 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.660236 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066"} err="failed to get container status \"e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066\": rpc error: code = NotFound desc = could not find container \"e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066\": container with ID starting with e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066 not found: ID does not exist" Nov 03 01:52:28.660268 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.660247 5463 scope.go:122] "RemoveContainer" containerID="ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031" Nov 03 01:52:28.660457 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.660439 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031"} err="failed to get container status \"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031\": rpc error: code = NotFound desc = could not find container \"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031\": container with ID starting with ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031 not found: ID does not exist" Nov 03 01:52:28.660523 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.660458 5463 scope.go:122] "RemoveContainer" containerID="de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea" Nov 03 01:52:28.660661 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.660642 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea"} err="failed to get container status \"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea\": rpc error: code = NotFound desc = could not find container \"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea\": container with ID starting with de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea not found: ID does not exist" Nov 03 01:52:28.660707 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.660661 5463 scope.go:122] "RemoveContainer" containerID="be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6" Nov 03 01:52:28.660835 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.660817 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6"} err="failed to get container status \"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6\": rpc error: code = NotFound desc = could not find container \"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6\": container with ID starting with be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6 not found: ID does not exist" Nov 03 01:52:28.660896 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.660835 5463 scope.go:122] "RemoveContainer" containerID="8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447" Nov 03 01:52:28.661035 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.661016 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447"} err="failed to get container status \"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447\": rpc error: code = NotFound desc = could not find container \"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447\": container with ID starting with 8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447 not found: ID does not exist" Nov 03 01:52:28.661083 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.661037 5463 scope.go:122] "RemoveContainer" containerID="2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965" Nov 03 01:52:28.661227 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.661210 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965"} err="failed to get container status \"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965\": rpc error: code = NotFound desc = could not find container \"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965\": container with ID starting with 2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965 not found: ID does not exist" Nov 03 01:52:28.661297 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.661228 5463 scope.go:122] "RemoveContainer" containerID="03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b" Nov 03 01:52:28.661465 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.661448 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b"} err="failed to get container status \"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b\": rpc error: code = NotFound desc = could not find container \"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b\": container with ID starting with 03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b not found: ID does not exist" Nov 03 01:52:28.661511 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.661464 5463 scope.go:122] "RemoveContainer" containerID="e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066" Nov 03 01:52:28.661694 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.661676 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066"} err="failed to get container status \"e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066\": rpc error: code = NotFound desc = could not find container \"e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066\": container with ID starting with e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066 not found: ID does not exist" Nov 03 01:52:28.661742 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.661694 5463 scope.go:122] "RemoveContainer" containerID="ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031" Nov 03 01:52:28.661869 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.661852 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031"} err="failed to get container status \"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031\": rpc error: code = NotFound desc = could not find container \"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031\": container with ID starting with ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031 not found: ID does not exist" Nov 03 01:52:28.661934 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.661870 5463 scope.go:122] "RemoveContainer" containerID="de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea" Nov 03 01:52:28.662088 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.662071 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea"} err="failed to get container status \"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea\": rpc error: code = NotFound desc = could not find container \"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea\": container with ID starting with de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea not found: ID does not exist" Nov 03 01:52:28.662165 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.662089 5463 scope.go:122] "RemoveContainer" containerID="be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6" Nov 03 01:52:28.662386 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.662366 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6"} err="failed to get container status \"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6\": rpc error: code = NotFound desc = could not find container \"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6\": container with ID starting with be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6 not found: ID does not exist" Nov 03 01:52:28.662450 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.662386 5463 scope.go:122] "RemoveContainer" containerID="8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447" Nov 03 01:52:28.662641 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.662626 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447"} err="failed to get container status \"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447\": rpc error: code = NotFound desc = could not find container \"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447\": container with ID starting with 8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447 not found: ID does not exist" Nov 03 01:52:28.662641 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.662641 5463 scope.go:122] "RemoveContainer" containerID="2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965" Nov 03 01:52:28.662850 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.662832 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965"} err="failed to get container status \"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965\": rpc error: code = NotFound desc = could not find container \"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965\": container with ID starting with 2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965 not found: ID does not exist" Nov 03 01:52:28.662902 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.662851 5463 scope.go:122] "RemoveContainer" containerID="03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b" Nov 03 01:52:28.663057 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.663038 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b"} err="failed to get container status \"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b\": rpc error: code = NotFound desc = could not find container \"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b\": container with ID starting with 03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b not found: ID does not exist" Nov 03 01:52:28.663107 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.663058 5463 scope.go:122] "RemoveContainer" containerID="e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066" Nov 03 01:52:28.663243 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.663223 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066"} err="failed to get container status \"e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066\": rpc error: code = NotFound desc = could not find container \"e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066\": container with ID starting with e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066 not found: ID does not exist" Nov 03 01:52:28.663318 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.663245 5463 scope.go:122] "RemoveContainer" containerID="ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031" Nov 03 01:52:28.663495 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.663479 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031"} err="failed to get container status \"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031\": rpc error: code = NotFound desc = could not find container \"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031\": container with ID starting with ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031 not found: ID does not exist" Nov 03 01:52:28.663539 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.663495 5463 scope.go:122] "RemoveContainer" containerID="de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea" Nov 03 01:52:28.663694 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.663676 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea"} err="failed to get container status \"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea\": rpc error: code = NotFound desc = could not find container \"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea\": container with ID starting with de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea not found: ID does not exist" Nov 03 01:52:28.663761 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.663694 5463 scope.go:122] "RemoveContainer" containerID="be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6" Nov 03 01:52:28.663911 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.663894 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6"} err="failed to get container status \"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6\": rpc error: code = NotFound desc = could not find container \"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6\": container with ID starting with be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6 not found: ID does not exist" Nov 03 01:52:28.663955 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.663912 5463 scope.go:122] "RemoveContainer" containerID="8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447" Nov 03 01:52:28.664131 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.664112 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447"} err="failed to get container status \"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447\": rpc error: code = NotFound desc = could not find container \"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447\": container with ID starting with 8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447 not found: ID does not exist" Nov 03 01:52:28.664220 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.664132 5463 scope.go:122] "RemoveContainer" containerID="2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965" Nov 03 01:52:28.664416 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.664399 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965"} err="failed to get container status \"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965\": rpc error: code = NotFound desc = could not find container \"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965\": container with ID starting with 2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965 not found: ID does not exist" Nov 03 01:52:28.664462 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.664416 5463 scope.go:122] "RemoveContainer" containerID="03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b" Nov 03 01:52:28.664613 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.664596 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b"} err="failed to get container status \"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b\": rpc error: code = NotFound desc = could not find container \"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b\": container with ID starting with 03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b not found: ID does not exist" Nov 03 01:52:28.664677 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.664617 5463 scope.go:122] "RemoveContainer" containerID="e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066" Nov 03 01:52:28.664857 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.664838 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066"} err="failed to get container status \"e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066\": rpc error: code = NotFound desc = could not find container \"e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066\": container with ID starting with e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066 not found: ID does not exist" Nov 03 01:52:28.664911 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.664858 5463 scope.go:122] "RemoveContainer" containerID="ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031" Nov 03 01:52:28.665093 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.665072 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031"} err="failed to get container status \"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031\": rpc error: code = NotFound desc = could not find container \"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031\": container with ID starting with ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031 not found: ID does not exist" Nov 03 01:52:28.665093 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.665093 5463 scope.go:122] "RemoveContainer" containerID="de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea" Nov 03 01:52:28.665403 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.665383 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea"} err="failed to get container status \"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea\": rpc error: code = NotFound desc = could not find container \"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea\": container with ID starting with de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea not found: ID does not exist" Nov 03 01:52:28.665474 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.665404 5463 scope.go:122] "RemoveContainer" containerID="be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6" Nov 03 01:52:28.665657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.665637 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6"} err="failed to get container status \"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6\": rpc error: code = NotFound desc = could not find container \"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6\": container with ID starting with be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6 not found: ID does not exist" Nov 03 01:52:28.665717 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.665657 5463 scope.go:122] "RemoveContainer" containerID="8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447" Nov 03 01:52:28.665897 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.665876 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447"} err="failed to get container status \"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447\": rpc error: code = NotFound desc = could not find container \"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447\": container with ID starting with 8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447 not found: ID does not exist" Nov 03 01:52:28.665944 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.665899 5463 scope.go:122] "RemoveContainer" containerID="2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965" Nov 03 01:52:28.666150 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.666128 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965"} err="failed to get container status \"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965\": rpc error: code = NotFound desc = could not find container \"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965\": container with ID starting with 2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965 not found: ID does not exist" Nov 03 01:52:28.666150 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.666147 5463 scope.go:122] "RemoveContainer" containerID="03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b" Nov 03 01:52:28.666434 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.666415 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b"} err="failed to get container status \"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b\": rpc error: code = NotFound desc = could not find container \"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b\": container with ID starting with 03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b not found: ID does not exist" Nov 03 01:52:28.666509 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.666435 5463 scope.go:122] "RemoveContainer" containerID="e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066" Nov 03 01:52:28.666670 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.666647 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066"} err="failed to get container status \"e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066\": rpc error: code = NotFound desc = could not find container \"e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066\": container with ID starting with e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066 not found: ID does not exist" Nov 03 01:52:28.666715 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.666671 5463 scope.go:122] "RemoveContainer" containerID="ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031" Nov 03 01:52:28.666925 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.666909 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031"} err="failed to get container status \"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031\": rpc error: code = NotFound desc = could not find container \"ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031\": container with ID starting with ba5f2d1412f6f4d48f1e113eb11abea7486734a4ea2344eb35b2a9f9f46a8031 not found: ID does not exist" Nov 03 01:52:28.666988 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.666926 5463 scope.go:122] "RemoveContainer" containerID="de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea" Nov 03 01:52:28.667160 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.667142 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea"} err="failed to get container status \"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea\": rpc error: code = NotFound desc = could not find container \"de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea\": container with ID starting with de8f5fb8d6bcd1b65dd5a2eb4fd6306371f8e6c053106fdc6aff9760c73306ea not found: ID does not exist" Nov 03 01:52:28.667251 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.667161 5463 scope.go:122] "RemoveContainer" containerID="be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6" Nov 03 01:52:28.667453 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.667434 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6"} err="failed to get container status \"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6\": rpc error: code = NotFound desc = could not find container \"be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6\": container with ID starting with be84e7983af053e8e5325ff7e02438d65b9755488b52aead1d2b7cf064b65df6 not found: ID does not exist" Nov 03 01:52:28.667508 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.667453 5463 scope.go:122] "RemoveContainer" containerID="8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447" Nov 03 01:52:28.667716 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.667695 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447"} err="failed to get container status \"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447\": rpc error: code = NotFound desc = could not find container \"8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447\": container with ID starting with 8fb5ed6ea5e9bd9abf701af00a87bbbcf8de63d960b2551f8f8832a5b63c8447 not found: ID does not exist" Nov 03 01:52:28.667716 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.667715 5463 scope.go:122] "RemoveContainer" containerID="2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965" Nov 03 01:52:28.667969 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.667950 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965"} err="failed to get container status \"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965\": rpc error: code = NotFound desc = could not find container \"2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965\": container with ID starting with 2ab9690772e71b717102e47a337ca96f46c6c2ccc8a7a9e85c2a9889e8b0d965 not found: ID does not exist" Nov 03 01:52:28.668045 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.667970 5463 scope.go:122] "RemoveContainer" containerID="03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b" Nov 03 01:52:28.668258 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.668240 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b"} err="failed to get container status \"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b\": rpc error: code = NotFound desc = could not find container \"03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b\": container with ID starting with 03e7539961e89f81c3314b5e0c81629079aed472efbd39d81f6a7f164cd8484b not found: ID does not exist" Nov 03 01:52:28.668319 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.668259 5463 scope.go:122] "RemoveContainer" containerID="e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066" Nov 03 01:52:28.668495 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.668479 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066"} err="failed to get container status \"e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066\": rpc error: code = NotFound desc = could not find container \"e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066\": container with ID starting with e5f187a653fc99a0a68eb2c5e20a066e6a6a5404ea34d38d3c46824978a2e066 not found: ID does not exist" Nov 03 01:52:28.783865 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.783802 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 01:52:28.834113 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834087 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 01:52:28.834390 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834372 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 01:52:28.834618 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834607 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="thanos-sidecar" Nov 03 01:52:28.834653 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834619 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="thanos-sidecar" Nov 03 01:52:28.834653 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834629 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="kube-rbac-proxy-thanos" Nov 03 01:52:28.834653 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834635 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="kube-rbac-proxy-thanos" Nov 03 01:52:28.834653 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834651 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="kube-rbac-proxy-web" Nov 03 01:52:28.834768 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834656 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="kube-rbac-proxy-web" Nov 03 01:52:28.834768 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834663 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="init-config-reloader" Nov 03 01:52:28.834768 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834667 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="init-config-reloader" Nov 03 01:52:28.834768 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834679 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="kube-rbac-proxy" Nov 03 01:52:28.834768 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834683 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="kube-rbac-proxy" Nov 03 01:52:28.834768 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834690 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="config-reloader" Nov 03 01:52:28.834768 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834694 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="config-reloader" Nov 03 01:52:28.834768 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834702 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="prometheus" Nov 03 01:52:28.834768 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834707 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="prometheus" Nov 03 01:52:28.834768 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834743 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="kube-rbac-proxy" Nov 03 01:52:28.834768 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834751 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="config-reloader" Nov 03 01:52:28.834768 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834757 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="prometheus" Nov 03 01:52:28.834768 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834764 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="thanos-sidecar" Nov 03 01:52:28.834768 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834769 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="kube-rbac-proxy-thanos" Nov 03 01:52:28.835149 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.834777 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" containerName="kube-rbac-proxy-web" Nov 03 01:52:28.838407 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.838392 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.879004 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.878981 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.879112 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.879012 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.879112 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.879035 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13434445-adb8-4652-947e-91241aee3c81-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.879112 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.879083 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.879299 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.879120 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.879299 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.879154 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.879299 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.879290 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncrk5\" (UniqueName: \"kubernetes.io/projected/13434445-adb8-4652-947e-91241aee3c81-kube-api-access-ncrk5\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.879424 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.879345 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.879424 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.879374 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.879424 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.879403 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.879424 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.879420 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-web-config\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.879548 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.879441 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.879548 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.879457 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.879548 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.879475 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.879639 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.879550 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13434445-adb8-4652-947e-91241aee3c81-config-out\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.879639 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.879598 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.879639 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.879631 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.879736 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.879654 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13434445-adb8-4652-947e-91241aee3c81-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.879736 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.879669 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-config\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.879736 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.879684 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.892290 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.892266 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 01:52:28.892396 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.892379 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 01:52:28.892466 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.892399 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 01:52:28.892512 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.892484 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 01:52:28.892512 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.892461 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 01:52:28.892891 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.892720 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 01:52:28.892891 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.892714 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 01:52:28.892891 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.892793 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 01:52:28.892891 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.892838 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nj9dq\"" Nov 03 01:52:28.892891 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.892858 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 01:52:28.893379 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.893056 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 01:52:28.893379 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.893121 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 01:52:28.893379 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.893160 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 01:52:28.893379 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.893335 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8rsahm0n9sh3l\"" Nov 03 01:52:28.893602 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.893385 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 01:52:28.943847 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.943821 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 01:52:28.943847 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.943833 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 01:52:28.944455 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.944439 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 01:52:28.980751 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.980733 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13434445-adb8-4652-947e-91241aee3c81-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.980863 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.980757 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-config\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.980863 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.980773 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.980863 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.980790 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.980863 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.980807 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.980863 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.980827 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13434445-adb8-4652-947e-91241aee3c81-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.980863 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.980845 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.980863 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.980862 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.981175 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.981152 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.981273 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.981209 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13434445-adb8-4652-947e-91241aee3c81-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.981273 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.981222 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncrk5\" (UniqueName: \"kubernetes.io/projected/13434445-adb8-4652-947e-91241aee3c81-kube-api-access-ncrk5\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.981273 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.981238 5463 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 01:52:28.981421 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.981282 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.981421 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.981310 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.981421 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.981336 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.981421 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.981363 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-web-config\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.981421 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.981396 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.981654 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.981424 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.981654 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.981452 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.981654 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.981491 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13434445-adb8-4652-947e-91241aee3c81-config-out\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.981654 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.981517 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.981654 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.981540 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.981654 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.981644 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.981858 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.981783 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.982307 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.982278 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.982400 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.982318 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.982400 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.982388 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.983862 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.983842 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13434445-adb8-4652-947e-91241aee3c81-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.983988 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.983889 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.984127 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.984079 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.985180 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.985153 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.985857 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.985828 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.986043 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.986024 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-config\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.986403 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.986378 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.986692 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.986666 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-web-config\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.986791 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.986731 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13434445-adb8-4652-947e-91241aee3c81-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.986888 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.986865 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.987052 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.987031 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.987124 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.987105 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13434445-adb8-4652-947e-91241aee3c81-config-out\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:28.987512 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:28.987495 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13434445-adb8-4652-947e-91241aee3c81-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:29.038342 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:29.038297 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncrk5\" (UniqueName: \"kubernetes.io/projected/13434445-adb8-4652-947e-91241aee3c81-kube-api-access-ncrk5\") pod \"prometheus-k8s-0\" (UID: \"13434445-adb8-4652-947e-91241aee3c81\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:29.156500 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:29.156464 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:29.621028 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:29.620998 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 01:52:29.647293 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:52:29.647263 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13434445_adb8_4652_947e_91241aee3c81.slice/crio-07c8a4d65e94db3b64fa5cf32dcf597fef3430d961c862dd76f415cc531e812b WatchSource:0}: Error finding container 07c8a4d65e94db3b64fa5cf32dcf597fef3430d961c862dd76f415cc531e812b: Status 404 returned error can't find the container with id 07c8a4d65e94db3b64fa5cf32dcf597fef3430d961c862dd76f415cc531e812b Nov 03 01:52:29.654253 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:29.654226 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:29.804402 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:29.804358 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:30.440577 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:30.440547 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a134e24f-d1ad-467a-914e-0ff35ca0177b" path="/var/lib/kubelet/pods/a134e24f-d1ad-467a-914e-0ff35ca0177b/volumes" Nov 03 01:52:30.500875 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:30.500853 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rb2h2" Nov 03 01:52:30.627583 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:30.627556 5463 generic.go:356] "Generic (PLEG): container finished" podID="13434445-adb8-4652-947e-91241aee3c81" containerID="24f0b6d67d3a09f68c9e93fb9613c997fae86eaef431060a0a48172b5c707932" exitCode=0 Nov 03 01:52:30.627730 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:30.627632 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13434445-adb8-4652-947e-91241aee3c81","Type":"ContainerDied","Data":"24f0b6d67d3a09f68c9e93fb9613c997fae86eaef431060a0a48172b5c707932"} Nov 03 01:52:30.627730 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:30.627688 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13434445-adb8-4652-947e-91241aee3c81","Type":"ContainerStarted","Data":"07c8a4d65e94db3b64fa5cf32dcf597fef3430d961c862dd76f415cc531e812b"} Nov 03 01:52:31.634925 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:31.634897 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13434445-adb8-4652-947e-91241aee3c81","Type":"ContainerStarted","Data":"9cba49bc379536e9b78a12b1344746a582dd79e43c68cc721117f04f1688ab45"} Nov 03 01:52:31.635365 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:31.634955 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13434445-adb8-4652-947e-91241aee3c81","Type":"ContainerStarted","Data":"f644871cdc0baac345e79562d8c2285d2a4cdc8123015d327503cb547efdaf66"} Nov 03 01:52:31.635365 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:31.634981 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13434445-adb8-4652-947e-91241aee3c81","Type":"ContainerStarted","Data":"821fae2604a9b0ae37128ddb77a114fa396dc51bd5943c56665671a49aabf8be"} Nov 03 01:52:31.635365 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:31.635003 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13434445-adb8-4652-947e-91241aee3c81","Type":"ContainerStarted","Data":"37dea9513d2f2e88219edc7b4764b28b6b3261488e88b688fdcb34f909c458b1"} Nov 03 01:52:31.635365 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:31.635027 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13434445-adb8-4652-947e-91241aee3c81","Type":"ContainerStarted","Data":"5fc3a1cf24626c774e22b7affa2109f9a92fb0d4b14098f56996bbe16e2036db"} Nov 03 01:52:31.635365 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:31.635050 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13434445-adb8-4652-947e-91241aee3c81","Type":"ContainerStarted","Data":"82f6afea824599f644574e2714c78db4e988be94df31dac4f929a3c93ba8643a"} Nov 03 01:52:31.760097 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:31.760046 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.742444784 podStartE2EDuration="3.760035703s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.590919ms" isStatefulPod=false podCreationTimestamp="2026-11-03 01:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:52:31.759241724 +0000 UTC m=+33.443350875" watchObservedRunningTime="2026-11-03 01:52:31.760035703 +0000 UTC m=+33.444144859" Nov 03 01:52:34.157300 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:34.157268 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:43.004371 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:52:43.004338 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 01:52:44.660234 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:44.660186 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:44.809399 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:44.809364 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:47.821213 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:47.820459 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:47.840863 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:47.840832 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:52:48.710113 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:52:48.710089 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 01:53:03.900265 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:53:03.900229 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:13.004805 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:53:13.004769 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 01:53:43.005902 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:53:43.005824 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 01:54:04.208509 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:04.208478 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:13.004751 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:54:13.004711 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 01:54:19.742130 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:19.742094 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:19.934940 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:19.934911 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:21.137656 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.137620 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:21.296515 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.296479 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:21.605647 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.605613 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf"] Nov 03 01:54:21.607948 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.607926 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf" Nov 03 01:54:21.688084 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.688057 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d73d0e9c-28be-4181-bc8f-987d41d487f2-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf\" (UID: \"d73d0e9c-28be-4181-bc8f-987d41d487f2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf" Nov 03 01:54:21.688259 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.688098 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfjhb\" (UniqueName: \"kubernetes.io/projected/d73d0e9c-28be-4181-bc8f-987d41d487f2-kube-api-access-kfjhb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf\" (UID: \"d73d0e9c-28be-4181-bc8f-987d41d487f2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf" Nov 03 01:54:21.688335 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.688271 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d73d0e9c-28be-4181-bc8f-987d41d487f2-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf\" (UID: \"d73d0e9c-28be-4181-bc8f-987d41d487f2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf" Nov 03 01:54:21.711369 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.711344 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-dr55r\"" Nov 03 01:54:21.711502 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.711402 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 01:54:21.711502 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.711401 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 01:54:21.711883 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.711867 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf"] Nov 03 01:54:21.712292 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.712275 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch"] Nov 03 01:54:21.714564 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.714550 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch" Nov 03 01:54:21.763838 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.763813 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch"] Nov 03 01:54:21.768081 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.768061 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-jw44s\"" Nov 03 01:54:21.788722 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.788699 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d73d0e9c-28be-4181-bc8f-987d41d487f2-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf\" (UID: \"d73d0e9c-28be-4181-bc8f-987d41d487f2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf" Nov 03 01:54:21.788821 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.788737 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d73d0e9c-28be-4181-bc8f-987d41d487f2-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf\" (UID: \"d73d0e9c-28be-4181-bc8f-987d41d487f2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf" Nov 03 01:54:21.788821 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.788756 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfjhb\" (UniqueName: \"kubernetes.io/projected/d73d0e9c-28be-4181-bc8f-987d41d487f2-kube-api-access-kfjhb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf\" (UID: \"d73d0e9c-28be-4181-bc8f-987d41d487f2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf" Nov 03 01:54:21.788821 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.788785 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch\" (UID: \"c1f4153b-edb6-4e8d-93c5-c14eaa71e741\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch" Nov 03 01:54:21.788928 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.788820 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch\" (UID: \"c1f4153b-edb6-4e8d-93c5-c14eaa71e741\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch" Nov 03 01:54:21.788928 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.788862 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhtqn\" (UniqueName: \"kubernetes.io/projected/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-kube-api-access-vhtqn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch\" (UID: \"c1f4153b-edb6-4e8d-93c5-c14eaa71e741\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch" Nov 03 01:54:21.789074 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.789059 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d73d0e9c-28be-4181-bc8f-987d41d487f2-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf\" (UID: \"d73d0e9c-28be-4181-bc8f-987d41d487f2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf" Nov 03 01:54:21.789114 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.789093 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d73d0e9c-28be-4181-bc8f-987d41d487f2-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf\" (UID: \"d73d0e9c-28be-4181-bc8f-987d41d487f2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf" Nov 03 01:54:21.809822 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.809797 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk"] Nov 03 01:54:21.812388 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.812374 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk" Nov 03 01:54:21.846051 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.846030 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfjhb\" (UniqueName: \"kubernetes.io/projected/d73d0e9c-28be-4181-bc8f-987d41d487f2-kube-api-access-kfjhb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf\" (UID: \"d73d0e9c-28be-4181-bc8f-987d41d487f2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf" Nov 03 01:54:21.867765 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.867712 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk"] Nov 03 01:54:21.870563 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.870548 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-d6qwm\"" Nov 03 01:54:21.889493 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.889468 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk\" (UID: \"2b7ed792-f8d3-4777-9bd5-58eb728d2a24\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk" Nov 03 01:54:21.889579 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.889499 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm2zm\" (UniqueName: \"kubernetes.io/projected/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-kube-api-access-fm2zm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk\" (UID: \"2b7ed792-f8d3-4777-9bd5-58eb728d2a24\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk" Nov 03 01:54:21.889579 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.889532 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch\" (UID: \"c1f4153b-edb6-4e8d-93c5-c14eaa71e741\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch" Nov 03 01:54:21.889694 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.889678 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk\" (UID: \"2b7ed792-f8d3-4777-9bd5-58eb728d2a24\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk" Nov 03 01:54:21.889734 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.889707 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch\" (UID: \"c1f4153b-edb6-4e8d-93c5-c14eaa71e741\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch" Nov 03 01:54:21.889796 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.889780 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhtqn\" (UniqueName: \"kubernetes.io/projected/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-kube-api-access-vhtqn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch\" (UID: \"c1f4153b-edb6-4e8d-93c5-c14eaa71e741\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch" Nov 03 01:54:21.889847 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.889829 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch\" (UID: \"c1f4153b-edb6-4e8d-93c5-c14eaa71e741\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch" Nov 03 01:54:21.890023 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.890007 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch\" (UID: \"c1f4153b-edb6-4e8d-93c5-c14eaa71e741\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch" Nov 03 01:54:21.907521 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.907498 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr"] Nov 03 01:54:21.909903 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.909889 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr" Nov 03 01:54:21.920033 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.920015 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf" Nov 03 01:54:21.946167 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.946144 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhtqn\" (UniqueName: \"kubernetes.io/projected/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-kube-api-access-vhtqn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch\" (UID: \"c1f4153b-edb6-4e8d-93c5-c14eaa71e741\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch" Nov 03 01:54:21.964370 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.964342 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr"] Nov 03 01:54:21.968697 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.968637 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-f2486\"" Nov 03 01:54:21.990532 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.990500 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk\" (UID: \"2b7ed792-f8d3-4777-9bd5-58eb728d2a24\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk" Nov 03 01:54:21.990667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.990532 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm2zm\" (UniqueName: \"kubernetes.io/projected/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-kube-api-access-fm2zm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk\" (UID: \"2b7ed792-f8d3-4777-9bd5-58eb728d2a24\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk" Nov 03 01:54:21.990667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.990560 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb234da4-434c-4c86-9bbd-7f69281a5945-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr\" (UID: \"bb234da4-434c-4c86-9bbd-7f69281a5945\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr" Nov 03 01:54:21.990667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.990589 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb234da4-434c-4c86-9bbd-7f69281a5945-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr\" (UID: \"bb234da4-434c-4c86-9bbd-7f69281a5945\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr" Nov 03 01:54:21.990667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.990615 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c47wq\" (UniqueName: \"kubernetes.io/projected/bb234da4-434c-4c86-9bbd-7f69281a5945-kube-api-access-c47wq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr\" (UID: \"bb234da4-434c-4c86-9bbd-7f69281a5945\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr" Nov 03 01:54:21.990667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.990655 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk\" (UID: \"2b7ed792-f8d3-4777-9bd5-58eb728d2a24\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk" Nov 03 01:54:21.990925 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.990898 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk\" (UID: \"2b7ed792-f8d3-4777-9bd5-58eb728d2a24\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk" Nov 03 01:54:21.990979 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:21.990947 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk\" (UID: \"2b7ed792-f8d3-4777-9bd5-58eb728d2a24\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk" Nov 03 01:54:22.027045 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:22.027022 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch" Nov 03 01:54:22.049677 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:22.049656 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm2zm\" (UniqueName: \"kubernetes.io/projected/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-kube-api-access-fm2zm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk\" (UID: \"2b7ed792-f8d3-4777-9bd5-58eb728d2a24\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk" Nov 03 01:54:22.070788 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:22.070759 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:22.091717 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:22.091695 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb234da4-434c-4c86-9bbd-7f69281a5945-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr\" (UID: \"bb234da4-434c-4c86-9bbd-7f69281a5945\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr" Nov 03 01:54:22.091829 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:22.091726 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb234da4-434c-4c86-9bbd-7f69281a5945-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr\" (UID: \"bb234da4-434c-4c86-9bbd-7f69281a5945\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr" Nov 03 01:54:22.091829 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:22.091753 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c47wq\" (UniqueName: \"kubernetes.io/projected/bb234da4-434c-4c86-9bbd-7f69281a5945-kube-api-access-c47wq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr\" (UID: \"bb234da4-434c-4c86-9bbd-7f69281a5945\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr" Nov 03 01:54:22.092074 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:22.092056 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb234da4-434c-4c86-9bbd-7f69281a5945-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr\" (UID: \"bb234da4-434c-4c86-9bbd-7f69281a5945\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr" Nov 03 01:54:22.092147 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:22.092094 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb234da4-434c-4c86-9bbd-7f69281a5945-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr\" (UID: \"bb234da4-434c-4c86-9bbd-7f69281a5945\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr" Nov 03 01:54:22.124073 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:22.124025 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk" Nov 03 01:54:22.149042 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:22.149019 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c47wq\" (UniqueName: \"kubernetes.io/projected/bb234da4-434c-4c86-9bbd-7f69281a5945-kube-api-access-c47wq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr\" (UID: \"bb234da4-434c-4c86-9bbd-7f69281a5945\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr" Nov 03 01:54:22.222977 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:22.222949 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr" Nov 03 01:54:22.286275 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:22.286247 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf"] Nov 03 01:54:22.286753 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:54:22.286730 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73d0e9c_28be_4181_bc8f_987d41d487f2.slice/crio-39fe9aaae7d796fc9ed0ed9f731f37c53568d09e4937b041ddca1a1e0200759b WatchSource:0}: Error finding container 39fe9aaae7d796fc9ed0ed9f731f37c53568d09e4937b041ddca1a1e0200759b: Status 404 returned error can't find the container with id 39fe9aaae7d796fc9ed0ed9f731f37c53568d09e4937b041ddca1a1e0200759b Nov 03 01:54:22.391856 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:54:22.391824 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f4153b_edb6_4e8d_93c5_c14eaa71e741.slice/crio-ae4453e4b2a6e30a5f67f0c9d6d51b718c905905698a99d3b50e109319f6cd2f WatchSource:0}: Error finding container ae4453e4b2a6e30a5f67f0c9d6d51b718c905905698a99d3b50e109319f6cd2f: Status 404 returned error can't find the container with id ae4453e4b2a6e30a5f67f0c9d6d51b718c905905698a99d3b50e109319f6cd2f Nov 03 01:54:22.393326 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:22.393299 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch"] Nov 03 01:54:22.484414 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:54:22.484380 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7ed792_f8d3_4777_9bd5_58eb728d2a24.slice/crio-ee9784888bc4626d3aa48065f085ae306b3eae655239aaa95676110b4678c1c0 WatchSource:0}: Error finding container ee9784888bc4626d3aa48065f085ae306b3eae655239aaa95676110b4678c1c0: Status 404 returned error can't find the container with id ee9784888bc4626d3aa48065f085ae306b3eae655239aaa95676110b4678c1c0 Nov 03 01:54:22.484582 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:22.484559 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk"] Nov 03 01:54:22.583282 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:54:22.583256 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb234da4_434c_4c86_9bbd_7f69281a5945.slice/crio-a8e5e6159d7fea58e947bf36a5915513e1b3c7e4acb2d7dba66f483ce851984e WatchSource:0}: Error finding container a8e5e6159d7fea58e947bf36a5915513e1b3c7e4acb2d7dba66f483ce851984e: Status 404 returned error can't find the container with id a8e5e6159d7fea58e947bf36a5915513e1b3c7e4acb2d7dba66f483ce851984e Nov 03 01:54:22.584837 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:22.584781 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr"] Nov 03 01:54:23.017165 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:23.017125 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr" event={"ID":"bb234da4-434c-4c86-9bbd-7f69281a5945","Type":"ContainerStarted","Data":"a8e5e6159d7fea58e947bf36a5915513e1b3c7e4acb2d7dba66f483ce851984e"} Nov 03 01:54:23.018313 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:23.018289 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk" event={"ID":"2b7ed792-f8d3-4777-9bd5-58eb728d2a24","Type":"ContainerStarted","Data":"ee9784888bc4626d3aa48065f085ae306b3eae655239aaa95676110b4678c1c0"} Nov 03 01:54:23.019364 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:23.019344 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf" event={"ID":"d73d0e9c-28be-4181-bc8f-987d41d487f2","Type":"ContainerStarted","Data":"39fe9aaae7d796fc9ed0ed9f731f37c53568d09e4937b041ddca1a1e0200759b"} Nov 03 01:54:23.020369 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:23.020351 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch" event={"ID":"c1f4153b-edb6-4e8d-93c5-c14eaa71e741","Type":"ContainerStarted","Data":"ae4453e4b2a6e30a5f67f0c9d6d51b718c905905698a99d3b50e109319f6cd2f"} Nov 03 01:54:24.101162 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.099345 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv"] Nov 03 01:54:24.102855 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.102378 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv" Nov 03 01:54:24.156920 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.156878 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv"] Nov 03 01:54:24.206213 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.206150 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-llzfx\"" Nov 03 01:54:24.211807 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.211698 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv\" (UID: \"ea166405-eaf6-4a1d-b09e-7333bc94b8f7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv" Nov 03 01:54:24.211807 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.211773 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rmbc\" (UniqueName: \"kubernetes.io/projected/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-kube-api-access-8rmbc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv\" (UID: \"ea166405-eaf6-4a1d-b09e-7333bc94b8f7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv" Nov 03 01:54:24.212027 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.211880 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv\" (UID: \"ea166405-eaf6-4a1d-b09e-7333bc94b8f7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv" Nov 03 01:54:24.313348 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.313308 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rmbc\" (UniqueName: \"kubernetes.io/projected/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-kube-api-access-8rmbc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv\" (UID: \"ea166405-eaf6-4a1d-b09e-7333bc94b8f7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv" Nov 03 01:54:24.313526 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.313391 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv\" (UID: \"ea166405-eaf6-4a1d-b09e-7333bc94b8f7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv" Nov 03 01:54:24.313593 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.313549 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv\" (UID: \"ea166405-eaf6-4a1d-b09e-7333bc94b8f7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv" Nov 03 01:54:24.314163 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.314133 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv\" (UID: \"ea166405-eaf6-4a1d-b09e-7333bc94b8f7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv" Nov 03 01:54:24.314539 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.314515 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv\" (UID: \"ea166405-eaf6-4a1d-b09e-7333bc94b8f7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv" Nov 03 01:54:24.376469 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.376405 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rmbc\" (UniqueName: \"kubernetes.io/projected/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-kube-api-access-8rmbc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv\" (UID: \"ea166405-eaf6-4a1d-b09e-7333bc94b8f7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv" Nov 03 01:54:24.420185 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.420154 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv" Nov 03 01:54:24.592277 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.592244 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp"] Nov 03 01:54:24.596526 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.596501 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp" Nov 03 01:54:24.647734 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.647683 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp"] Nov 03 01:54:24.649486 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.649461 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-4wxm2\"" Nov 03 01:54:24.717095 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.717060 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw9t4\" (UniqueName: \"kubernetes.io/projected/cd0202fb-1f79-404b-b86c-a53d22961e10-kube-api-access-cw9t4\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp\" (UID: \"cd0202fb-1f79-404b-b86c-a53d22961e10\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp" Nov 03 01:54:24.717275 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.717148 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd0202fb-1f79-404b-b86c-a53d22961e10-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp\" (UID: \"cd0202fb-1f79-404b-b86c-a53d22961e10\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp" Nov 03 01:54:24.717524 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.717337 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd0202fb-1f79-404b-b86c-a53d22961e10-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp\" (UID: \"cd0202fb-1f79-404b-b86c-a53d22961e10\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp" Nov 03 01:54:24.813921 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.813887 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv"] Nov 03 01:54:24.823281 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:54:24.820434 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea166405_eaf6_4a1d_b09e_7333bc94b8f7.slice/crio-f1b928dc6327226b3a24df31953ec75295ae603926c2299f72a08e19d22d4cd9 WatchSource:0}: Error finding container f1b928dc6327226b3a24df31953ec75295ae603926c2299f72a08e19d22d4cd9: Status 404 returned error can't find the container with id f1b928dc6327226b3a24df31953ec75295ae603926c2299f72a08e19d22d4cd9 Nov 03 01:54:24.823281 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.821758 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw9t4\" (UniqueName: \"kubernetes.io/projected/cd0202fb-1f79-404b-b86c-a53d22961e10-kube-api-access-cw9t4\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp\" (UID: \"cd0202fb-1f79-404b-b86c-a53d22961e10\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp" Nov 03 01:54:24.823281 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.821835 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd0202fb-1f79-404b-b86c-a53d22961e10-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp\" (UID: \"cd0202fb-1f79-404b-b86c-a53d22961e10\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp" Nov 03 01:54:24.823281 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.821907 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd0202fb-1f79-404b-b86c-a53d22961e10-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp\" (UID: \"cd0202fb-1f79-404b-b86c-a53d22961e10\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp" Nov 03 01:54:24.823281 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.822683 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd0202fb-1f79-404b-b86c-a53d22961e10-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp\" (UID: \"cd0202fb-1f79-404b-b86c-a53d22961e10\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp" Nov 03 01:54:24.823281 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.822788 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd0202fb-1f79-404b-b86c-a53d22961e10-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp\" (UID: \"cd0202fb-1f79-404b-b86c-a53d22961e10\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp" Nov 03 01:54:24.883454 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.883421 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw9t4\" (UniqueName: \"kubernetes.io/projected/cd0202fb-1f79-404b-b86c-a53d22961e10-kube-api-access-cw9t4\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp\" (UID: \"cd0202fb-1f79-404b-b86c-a53d22961e10\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp" Nov 03 01:54:24.915156 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:24.915088 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp" Nov 03 01:54:25.040395 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:25.040361 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv" event={"ID":"ea166405-eaf6-4a1d-b09e-7333bc94b8f7","Type":"ContainerStarted","Data":"f1b928dc6327226b3a24df31953ec75295ae603926c2299f72a08e19d22d4cd9"} Nov 03 01:54:25.315172 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:25.315145 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp"] Nov 03 01:54:25.316154 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:54:25.316126 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0202fb_1f79_404b_b86c_a53d22961e10.slice/crio-c69c79a802e89b86797aaec31d1c1ccd865d6a2af7ee0b71398d5e3c708a8c11 WatchSource:0}: Error finding container c69c79a802e89b86797aaec31d1c1ccd865d6a2af7ee0b71398d5e3c708a8c11: Status 404 returned error can't find the container with id c69c79a802e89b86797aaec31d1c1ccd865d6a2af7ee0b71398d5e3c708a8c11 Nov 03 01:54:26.048206 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:26.048134 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp" event={"ID":"cd0202fb-1f79-404b-b86c-a53d22961e10","Type":"ContainerStarted","Data":"c69c79a802e89b86797aaec31d1c1ccd865d6a2af7ee0b71398d5e3c708a8c11"} Nov 03 01:54:27.708593 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:27.708551 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n"] Nov 03 01:54:27.714085 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:27.714063 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n" Nov 03 01:54:27.766131 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:27.766099 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n"] Nov 03 01:54:27.815080 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:27.815056 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-qfgzh\"" Nov 03 01:54:27.856906 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:27.856873 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkgdp\" (UniqueName: \"kubernetes.io/projected/14260093-edd8-4d31-abe2-9ce1b3947977-kube-api-access-qkgdp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n\" (UID: \"14260093-edd8-4d31-abe2-9ce1b3947977\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n" Nov 03 01:54:27.857045 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:27.856919 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14260093-edd8-4d31-abe2-9ce1b3947977-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n\" (UID: \"14260093-edd8-4d31-abe2-9ce1b3947977\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n" Nov 03 01:54:27.857090 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:27.857047 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14260093-edd8-4d31-abe2-9ce1b3947977-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n\" (UID: \"14260093-edd8-4d31-abe2-9ce1b3947977\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n" Nov 03 01:54:27.959149 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:27.957827 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14260093-edd8-4d31-abe2-9ce1b3947977-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n\" (UID: \"14260093-edd8-4d31-abe2-9ce1b3947977\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n" Nov 03 01:54:27.959149 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:27.957896 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkgdp\" (UniqueName: \"kubernetes.io/projected/14260093-edd8-4d31-abe2-9ce1b3947977-kube-api-access-qkgdp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n\" (UID: \"14260093-edd8-4d31-abe2-9ce1b3947977\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n" Nov 03 01:54:27.959149 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:27.957925 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14260093-edd8-4d31-abe2-9ce1b3947977-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n\" (UID: \"14260093-edd8-4d31-abe2-9ce1b3947977\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n" Nov 03 01:54:27.959149 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:27.958389 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14260093-edd8-4d31-abe2-9ce1b3947977-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n\" (UID: \"14260093-edd8-4d31-abe2-9ce1b3947977\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n" Nov 03 01:54:27.959149 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:27.958764 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14260093-edd8-4d31-abe2-9ce1b3947977-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n\" (UID: \"14260093-edd8-4d31-abe2-9ce1b3947977\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n" Nov 03 01:54:28.026372 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:28.026290 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkgdp\" (UniqueName: \"kubernetes.io/projected/14260093-edd8-4d31-abe2-9ce1b3947977-kube-api-access-qkgdp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n\" (UID: \"14260093-edd8-4d31-abe2-9ce1b3947977\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n" Nov 03 01:54:28.038537 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:28.038312 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n" Nov 03 01:54:28.456612 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:28.456588 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n"] Nov 03 01:54:29.068420 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:29.068381 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n" event={"ID":"14260093-edd8-4d31-abe2-9ce1b3947977","Type":"ContainerStarted","Data":"42bf727a131a489ab12f65318c52b6ddf134320d08ccbf39b64c59306809941b"} Nov 03 01:54:30.112097 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:30.112052 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:30.264957 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:30.264918 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:31.053062 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.053028 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt"] Nov 03 01:54:31.058336 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.058311 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.112264 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.112236 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 01:54:31.132004 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.131975 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 01:54:31.132151 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.132074 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fwpk8\"" Nov 03 01:54:31.161208 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.161166 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:31.162063 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.162016 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 01:54:31.163109 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.163086 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt"] Nov 03 01:54:31.206622 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.206581 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.206788 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.206631 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83d426e2-ed63-493c-871e-aa5d62476d77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.206788 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.206667 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.206788 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.206695 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.206788 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.206719 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83d426e2-ed63-493c-871e-aa5d62476d77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.206788 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.206747 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.206788 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.206774 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.207136 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.206804 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h6d9\" (UniqueName: \"kubernetes.io/projected/83d426e2-ed63-493c-871e-aa5d62476d77-kube-api-access-9h6d9\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.207136 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.206846 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83d426e2-ed63-493c-871e-aa5d62476d77-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.308046 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.307971 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.308046 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.308025 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.308403 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.308058 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83d426e2-ed63-493c-871e-aa5d62476d77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.308403 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.308089 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.308403 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.308116 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.308403 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.308148 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h6d9\" (UniqueName: \"kubernetes.io/projected/83d426e2-ed63-493c-871e-aa5d62476d77-kube-api-access-9h6d9\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.308403 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.308216 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83d426e2-ed63-493c-871e-aa5d62476d77-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.308403 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.308315 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.308403 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.308339 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83d426e2-ed63-493c-871e-aa5d62476d77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.311449 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.309111 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83d426e2-ed63-493c-871e-aa5d62476d77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.311449 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.309431 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.311449 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.309683 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.311449 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.310627 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.311449 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.310871 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.326617 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.326499 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.326891 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.326728 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83d426e2-ed63-493c-871e-aa5d62476d77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.358410 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.358347 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:31.370290 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.370223 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h6d9\" (UniqueName: \"kubernetes.io/projected/83d426e2-ed63-493c-871e-aa5d62476d77-kube-api-access-9h6d9\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.373105 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.373033 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83d426e2-ed63-493c-871e-aa5d62476d77-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m6hqt\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.379298 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.379103 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:31.506877 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.505423 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:31.841095 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:31.841064 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt"] Nov 03 01:54:31.843909 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:54:31.843883 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d426e2_ed63_493c_871e_aa5d62476d77.slice/crio-f8a8eede948885722349c5e99b6c0359dd2ae614fbc54e8d477c07d8a1d3f00a WatchSource:0}: Error finding container f8a8eede948885722349c5e99b6c0359dd2ae614fbc54e8d477c07d8a1d3f00a: Status 404 returned error can't find the container with id f8a8eede948885722349c5e99b6c0359dd2ae614fbc54e8d477c07d8a1d3f00a Nov 03 01:54:32.095436 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:32.095354 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" event={"ID":"83d426e2-ed63-493c-871e-aa5d62476d77","Type":"ContainerStarted","Data":"f8a8eede948885722349c5e99b6c0359dd2ae614fbc54e8d477c07d8a1d3f00a"} Nov 03 01:54:33.005586 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:33.005549 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:33.152655 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:33.152540 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:35.022893 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:35.022843 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:36.124822 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:36.124772 5463 generic.go:356] "Generic (PLEG): container finished" podID="bb234da4-434c-4c86-9bbd-7f69281a5945" containerID="c3bd2d4191546988bc3862eb4a91c1380e09cc3d05ad851ae60e2fb6654bdc08" exitCode=0 Nov 03 01:54:36.125278 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:36.124910 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr" event={"ID":"bb234da4-434c-4c86-9bbd-7f69281a5945","Type":"ContainerDied","Data":"c3bd2d4191546988bc3862eb4a91c1380e09cc3d05ad851ae60e2fb6654bdc08"} Nov 03 01:54:36.128789 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:36.128760 5463 generic.go:356] "Generic (PLEG): container finished" podID="d73d0e9c-28be-4181-bc8f-987d41d487f2" containerID="33094244ccad4f9cf09ae92b4260b1b238ac2ec0d9b824c51a31b37945d8e4fa" exitCode=0 Nov 03 01:54:36.128908 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:36.128828 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf" event={"ID":"d73d0e9c-28be-4181-bc8f-987d41d487f2","Type":"ContainerDied","Data":"33094244ccad4f9cf09ae92b4260b1b238ac2ec0d9b824c51a31b37945d8e4fa"} Nov 03 01:54:37.136105 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:37.136058 5463 generic.go:356] "Generic (PLEG): container finished" podID="2b7ed792-f8d3-4777-9bd5-58eb728d2a24" containerID="adad5eaf76a5665685d9bedf7ea46db0b4206a5c7608d47935ac23c65cead9be" exitCode=0 Nov 03 01:54:37.136590 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:37.136243 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk" event={"ID":"2b7ed792-f8d3-4777-9bd5-58eb728d2a24","Type":"ContainerDied","Data":"adad5eaf76a5665685d9bedf7ea46db0b4206a5c7608d47935ac23c65cead9be"} Nov 03 01:54:37.138852 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:37.138823 5463 generic.go:356] "Generic (PLEG): container finished" podID="c1f4153b-edb6-4e8d-93c5-c14eaa71e741" containerID="8fd05516c3c2a139fc3c67f80cff059828c01986a375b4dcbd7070e9b16553ce" exitCode=0 Nov 03 01:54:37.138958 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:37.138866 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch" event={"ID":"c1f4153b-edb6-4e8d-93c5-c14eaa71e741","Type":"ContainerDied","Data":"8fd05516c3c2a139fc3c67f80cff059828c01986a375b4dcbd7070e9b16553ce"} Nov 03 01:54:38.145079 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:38.145046 5463 generic.go:356] "Generic (PLEG): container finished" podID="14260093-edd8-4d31-abe2-9ce1b3947977" containerID="b50516006449bec6436236ce5983227e076a6a290e98f50778f7a26f6b063534" exitCode=0 Nov 03 01:54:38.145606 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:38.145139 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n" event={"ID":"14260093-edd8-4d31-abe2-9ce1b3947977","Type":"ContainerDied","Data":"b50516006449bec6436236ce5983227e076a6a290e98f50778f7a26f6b063534"} Nov 03 01:54:38.656352 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:38.656321 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:54:38.656452 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:38.656387 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:54:38.656452 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:38.656417 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:54:39.150737 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:39.150712 5463 generic.go:356] "Generic (PLEG): container finished" podID="bb234da4-434c-4c86-9bbd-7f69281a5945" containerID="d1606795998721a7106111b0c9561302f806ce2162f13a51f61daad60b1755d9" exitCode=0 Nov 03 01:54:39.151097 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:39.150799 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr" event={"ID":"bb234da4-434c-4c86-9bbd-7f69281a5945","Type":"ContainerDied","Data":"d1606795998721a7106111b0c9561302f806ce2162f13a51f61daad60b1755d9"} Nov 03 01:54:39.152396 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:39.152370 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" event={"ID":"83d426e2-ed63-493c-871e-aa5d62476d77","Type":"ContainerStarted","Data":"08d9a0691aa88b6a9ce41589b8d8fb008c907a3d2ae04c1de7d2c8d3a95871df"} Nov 03 01:54:39.154250 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:39.154233 5463 generic.go:356] "Generic (PLEG): container finished" podID="c1f4153b-edb6-4e8d-93c5-c14eaa71e741" containerID="74216defe0e6bbb3dc6c651a469ac554febabcb2d951e38961b90c873f22078d" exitCode=0 Nov 03 01:54:39.154353 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:39.154315 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch" event={"ID":"c1f4153b-edb6-4e8d-93c5-c14eaa71e741","Type":"ContainerDied","Data":"74216defe0e6bbb3dc6c651a469ac554febabcb2d951e38961b90c873f22078d"} Nov 03 01:54:39.156518 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:39.156487 5463 generic.go:356] "Generic (PLEG): container finished" podID="d73d0e9c-28be-4181-bc8f-987d41d487f2" containerID="7dd16eaee22c1d180e63e8b5e81f32ccda29f6c607bd08562350d3e6ec7948e6" exitCode=0 Nov 03 01:54:39.156601 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:39.156566 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf" event={"ID":"d73d0e9c-28be-4181-bc8f-987d41d487f2","Type":"ContainerDied","Data":"7dd16eaee22c1d180e63e8b5e81f32ccda29f6c607bd08562350d3e6ec7948e6"} Nov 03 01:54:39.158562 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:39.158235 5463 generic.go:356] "Generic (PLEG): container finished" podID="ea166405-eaf6-4a1d-b09e-7333bc94b8f7" containerID="cb85ebd48efe6b45722360fe4742e44671d37f5adcbe0e2a974669d58cf3a992" exitCode=0 Nov 03 01:54:39.158562 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:39.158318 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv" event={"ID":"ea166405-eaf6-4a1d-b09e-7333bc94b8f7","Type":"ContainerDied","Data":"cb85ebd48efe6b45722360fe4742e44671d37f5adcbe0e2a974669d58cf3a992"} Nov 03 01:54:39.160055 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:39.160036 5463 generic.go:356] "Generic (PLEG): container finished" podID="cd0202fb-1f79-404b-b86c-a53d22961e10" containerID="8c2d35d02a9af4537137ebd60e7409292607b4f9ff3cfc9a854b1e10c2bce0e4" exitCode=0 Nov 03 01:54:39.160100 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:39.160068 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp" event={"ID":"cd0202fb-1f79-404b-b86c-a53d22961e10","Type":"ContainerDied","Data":"8c2d35d02a9af4537137ebd60e7409292607b4f9ff3cfc9a854b1e10c2bce0e4"} Nov 03 01:54:39.379379 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:39.379347 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:39.384120 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:39.384089 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:39.511491 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:39.511401 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" podStartSLOduration=1.706708119 podStartE2EDuration="8.511387015s" totalImagesPullingTime="6.804678896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:54:39.510274462 +0000 UTC m=+161.194383616" watchObservedRunningTime="2026-11-03 01:54:39.511387015 +0000 UTC m=+161.195496169" Nov 03 01:54:40.175263 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:40.175073 5463 generic.go:356] "Generic (PLEG): container finished" podID="2b7ed792-f8d3-4777-9bd5-58eb728d2a24" containerID="a12bfe7b17d7f00fcc5ec5aab82175e4b7f7cbb2a2294f8ed763e0f7436bbc15" exitCode=0 Nov 03 01:54:40.177275 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:40.177247 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk" event={"ID":"2b7ed792-f8d3-4777-9bd5-58eb728d2a24","Type":"ContainerDied","Data":"a12bfe7b17d7f00fcc5ec5aab82175e4b7f7cbb2a2294f8ed763e0f7436bbc15"} Nov 03 01:54:40.177409 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:40.177301 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:40.178456 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:40.178435 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:54:41.185945 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:41.185910 5463 generic.go:356] "Generic (PLEG): container finished" podID="cd0202fb-1f79-404b-b86c-a53d22961e10" containerID="08f58f22636570154ce095782808969ae441b0ebae41a3f405d5cec8cd350873" exitCode=0 Nov 03 01:54:41.188176 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:41.187117 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp" event={"ID":"cd0202fb-1f79-404b-b86c-a53d22961e10","Type":"ContainerDied","Data":"08f58f22636570154ce095782808969ae441b0ebae41a3f405d5cec8cd350873"} Nov 03 01:54:42.199355 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:42.199283 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n" event={"ID":"14260093-edd8-4d31-abe2-9ce1b3947977","Type":"ContainerStarted","Data":"f8afbd9f70a127d0a2c811fe78520a3c62ccbac1dd238c753ff16040adbffb70"} Nov 03 01:54:43.020340 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:54:43.020304 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 01:54:43.213107 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:43.213063 5463 generic.go:356] "Generic (PLEG): container finished" podID="14260093-edd8-4d31-abe2-9ce1b3947977" containerID="f8afbd9f70a127d0a2c811fe78520a3c62ccbac1dd238c753ff16040adbffb70" exitCode=0 Nov 03 01:54:43.213704 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:43.213185 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n" event={"ID":"14260093-edd8-4d31-abe2-9ce1b3947977","Type":"ContainerDied","Data":"f8afbd9f70a127d0a2c811fe78520a3c62ccbac1dd238c753ff16040adbffb70"} Nov 03 01:54:46.237850 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:46.237812 5463 generic.go:356] "Generic (PLEG): container finished" podID="ea166405-eaf6-4a1d-b09e-7333bc94b8f7" containerID="3ae7ed8f30b9a1f9de52b193e2ce5cb141901a84c5f4e7ea88138b4cf86dee2a" exitCode=0 Nov 03 01:54:46.237850 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:46.237851 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv" event={"ID":"ea166405-eaf6-4a1d-b09e-7333bc94b8f7","Type":"ContainerDied","Data":"3ae7ed8f30b9a1f9de52b193e2ce5cb141901a84c5f4e7ea88138b4cf86dee2a"} Nov 03 01:54:54.302650 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:54.302606 5463 generic.go:356] "Generic (PLEG): container finished" podID="bb234da4-434c-4c86-9bbd-7f69281a5945" containerID="3b1caa76eadfde940bd8f258fd3ef42ee3e1860799037a9a43dc600706dcf3c3" exitCode=0 Nov 03 01:54:54.303093 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:54.302764 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr" event={"ID":"bb234da4-434c-4c86-9bbd-7f69281a5945","Type":"ContainerDied","Data":"3b1caa76eadfde940bd8f258fd3ef42ee3e1860799037a9a43dc600706dcf3c3"} Nov 03 01:54:54.306686 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:54.306658 5463 generic.go:356] "Generic (PLEG): container finished" podID="c1f4153b-edb6-4e8d-93c5-c14eaa71e741" containerID="ca552971ba1052556487d7174e992c838494b778f80c8443b4b97f2c12e0f7d6" exitCode=0 Nov 03 01:54:54.306797 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:54.306752 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch" event={"ID":"c1f4153b-edb6-4e8d-93c5-c14eaa71e741","Type":"ContainerDied","Data":"ca552971ba1052556487d7174e992c838494b778f80c8443b4b97f2c12e0f7d6"} Nov 03 01:54:54.311971 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:54.311334 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf" event={"ID":"d73d0e9c-28be-4181-bc8f-987d41d487f2","Type":"ContainerStarted","Data":"bb3a443076a9ad1a4e176f443a9e3385243ca7e17aece238993f0f0e53eceb39"} Nov 03 01:54:54.648756 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:54.648687 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf" podStartSLOduration=3.252154913 podStartE2EDuration="33.648667302s" totalImagesPullingTime="30.396512389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:54:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:54:54.647566593 +0000 UTC m=+176.331675749" watchObservedRunningTime="2026-11-03 01:54:54.648667302 +0000 UTC m=+176.332776454" Nov 03 01:54:55.317992 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.317959 5463 generic.go:356] "Generic (PLEG): container finished" podID="d73d0e9c-28be-4181-bc8f-987d41d487f2" containerID="bb3a443076a9ad1a4e176f443a9e3385243ca7e17aece238993f0f0e53eceb39" exitCode=0 Nov 03 01:54:55.318389 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.318046 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf" event={"ID":"d73d0e9c-28be-4181-bc8f-987d41d487f2","Type":"ContainerDied","Data":"bb3a443076a9ad1a4e176f443a9e3385243ca7e17aece238993f0f0e53eceb39"} Nov 03 01:54:55.320862 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.320575 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv" event={"ID":"ea166405-eaf6-4a1d-b09e-7333bc94b8f7","Type":"ContainerStarted","Data":"c1f2b5d5561b8cf72be5ce449642f3c6d72142f9a1ea5194ae698529a4395887"} Nov 03 01:54:55.324221 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.323794 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk" event={"ID":"2b7ed792-f8d3-4777-9bd5-58eb728d2a24","Type":"ContainerStarted","Data":"18d6f46a87027e0722e38963c243588ca38fdcf7d170fad18564fab22d0cfcf9"} Nov 03 01:54:55.492125 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.488450 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch" Nov 03 01:54:55.542938 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.542907 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-kube-api-access-vhtqn\" (UniqueName: \"kubernetes.io/projected/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-kube-api-access-vhtqn\") pod \"c1f4153b-edb6-4e8d-93c5-c14eaa71e741\" (UID: \"c1f4153b-edb6-4e8d-93c5-c14eaa71e741\") " Nov 03 01:54:55.543086 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.542995 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-util\" (UniqueName: \"kubernetes.io/empty-dir/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-util\") pod \"c1f4153b-edb6-4e8d-93c5-c14eaa71e741\" (UID: \"c1f4153b-edb6-4e8d-93c5-c14eaa71e741\") " Nov 03 01:54:55.543086 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.543070 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-bundle\") pod \"c1f4153b-edb6-4e8d-93c5-c14eaa71e741\" (UID: \"c1f4153b-edb6-4e8d-93c5-c14eaa71e741\") " Nov 03 01:54:55.543971 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.543939 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-bundle" pod "c1f4153b-edb6-4e8d-93c5-c14eaa71e741" (UID: "c1f4153b-edb6-4e8d-93c5-c14eaa71e741"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:55.545684 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.545659 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-kube-api-access-vhtqn" pod "c1f4153b-edb6-4e8d-93c5-c14eaa71e741" (UID: "c1f4153b-edb6-4e8d-93c5-c14eaa71e741"). InnerVolumeSpecName "kube-api-access-vhtqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:54:55.548214 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.548175 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-util" pod "c1f4153b-edb6-4e8d-93c5-c14eaa71e741" (UID: "c1f4153b-edb6-4e8d-93c5-c14eaa71e741"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:55.551229 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.551167 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk" podStartSLOduration=4.261143364 podStartE2EDuration="34.551151855s" totalImagesPullingTime="30.290008491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:54:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:54:55.549802936 +0000 UTC m=+177.233912092" watchObservedRunningTime="2026-11-03 01:54:55.551151855 +0000 UTC m=+177.235261025" Nov 03 01:54:55.574082 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.574049 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr" Nov 03 01:54:55.643607 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.643574 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb234da4-434c-4c86-9bbd-7f69281a5945-kube-api-access-c47wq\" (UniqueName: \"kubernetes.io/projected/bb234da4-434c-4c86-9bbd-7f69281a5945-kube-api-access-c47wq\") pod \"bb234da4-434c-4c86-9bbd-7f69281a5945\" (UID: \"bb234da4-434c-4c86-9bbd-7f69281a5945\") " Nov 03 01:54:55.643788 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.643770 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb234da4-434c-4c86-9bbd-7f69281a5945-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb234da4-434c-4c86-9bbd-7f69281a5945-bundle\") pod \"bb234da4-434c-4c86-9bbd-7f69281a5945\" (UID: \"bb234da4-434c-4c86-9bbd-7f69281a5945\") " Nov 03 01:54:55.643869 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.643827 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb234da4-434c-4c86-9bbd-7f69281a5945-util\" (UniqueName: \"kubernetes.io/empty-dir/bb234da4-434c-4c86-9bbd-7f69281a5945-util\") pod \"bb234da4-434c-4c86-9bbd-7f69281a5945\" (UID: \"bb234da4-434c-4c86-9bbd-7f69281a5945\") " Nov 03 01:54:55.644084 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.644047 5463 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-util\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:55.644209 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.644086 5463 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-bundle\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:55.644209 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.644102 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhtqn\" (UniqueName: \"kubernetes.io/projected/c1f4153b-edb6-4e8d-93c5-c14eaa71e741-kube-api-access-vhtqn\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:55.644374 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.644349 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb234da4-434c-4c86-9bbd-7f69281a5945-bundle" pod "bb234da4-434c-4c86-9bbd-7f69281a5945" (UID: "bb234da4-434c-4c86-9bbd-7f69281a5945"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:55.645916 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.645882 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb234da4-434c-4c86-9bbd-7f69281a5945-kube-api-access-c47wq" pod "bb234da4-434c-4c86-9bbd-7f69281a5945" (UID: "bb234da4-434c-4c86-9bbd-7f69281a5945"). InnerVolumeSpecName "kube-api-access-c47wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:54:55.648759 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.648736 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb234da4-434c-4c86-9bbd-7f69281a5945-util" pod "bb234da4-434c-4c86-9bbd-7f69281a5945" (UID: "bb234da4-434c-4c86-9bbd-7f69281a5945"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:55.665707 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.665667 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv" podStartSLOduration=2.596676071 podStartE2EDuration="31.665654233s" totalImagesPullingTime="29.068978162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:54:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:54:55.664204194 +0000 UTC m=+177.348313372" watchObservedRunningTime="2026-11-03 01:54:55.665654233 +0000 UTC m=+177.349763387" Nov 03 01:54:55.745057 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.744998 5463 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb234da4-434c-4c86-9bbd-7f69281a5945-util\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:55.745057 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.745021 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c47wq\" (UniqueName: \"kubernetes.io/projected/bb234da4-434c-4c86-9bbd-7f69281a5945-kube-api-access-c47wq\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:55.745057 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:55.745035 5463 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb234da4-434c-4c86-9bbd-7f69281a5945-bundle\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:56.329046 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.328975 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr" Nov 03 01:54:56.329046 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.329005 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75fvwr" event={"ID":"bb234da4-434c-4c86-9bbd-7f69281a5945","Type":"ContainerDied","Data":"a8e5e6159d7fea58e947bf36a5915513e1b3c7e4acb2d7dba66f483ce851984e"} Nov 03 01:54:56.329518 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.329055 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8e5e6159d7fea58e947bf36a5915513e1b3c7e4acb2d7dba66f483ce851984e" Nov 03 01:54:56.330919 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.330894 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch" Nov 03 01:54:56.330919 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.330905 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avbpch" event={"ID":"c1f4153b-edb6-4e8d-93c5-c14eaa71e741","Type":"ContainerDied","Data":"ae4453e4b2a6e30a5f67f0c9d6d51b718c905905698a99d3b50e109319f6cd2f"} Nov 03 01:54:56.331117 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.330939 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae4453e4b2a6e30a5f67f0c9d6d51b718c905905698a99d3b50e109319f6cd2f" Nov 03 01:54:56.333282 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.333258 5463 generic.go:356] "Generic (PLEG): container finished" podID="ea166405-eaf6-4a1d-b09e-7333bc94b8f7" containerID="c1f2b5d5561b8cf72be5ce449642f3c6d72142f9a1ea5194ae698529a4395887" exitCode=0 Nov 03 01:54:56.333367 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.333336 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv" event={"ID":"ea166405-eaf6-4a1d-b09e-7333bc94b8f7","Type":"ContainerDied","Data":"c1f2b5d5561b8cf72be5ce449642f3c6d72142f9a1ea5194ae698529a4395887"} Nov 03 01:54:56.335371 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.335346 5463 generic.go:356] "Generic (PLEG): container finished" podID="2b7ed792-f8d3-4777-9bd5-58eb728d2a24" containerID="18d6f46a87027e0722e38963c243588ca38fdcf7d170fad18564fab22d0cfcf9" exitCode=0 Nov 03 01:54:56.335468 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.335384 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk" event={"ID":"2b7ed792-f8d3-4777-9bd5-58eb728d2a24","Type":"ContainerDied","Data":"18d6f46a87027e0722e38963c243588ca38fdcf7d170fad18564fab22d0cfcf9"} Nov 03 01:54:56.337557 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.337535 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp" event={"ID":"cd0202fb-1f79-404b-b86c-a53d22961e10","Type":"ContainerStarted","Data":"e833c480cb90c21286a0b32576335d6a9c5326cadd63e36f43e8ce0363cdbe1f"} Nov 03 01:54:56.484796 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.484776 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf" Nov 03 01:54:56.552169 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.552134 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73d0e9c-28be-4181-bc8f-987d41d487f2-kube-api-access-kfjhb\" (UniqueName: \"kubernetes.io/projected/d73d0e9c-28be-4181-bc8f-987d41d487f2-kube-api-access-kfjhb\") pod \"d73d0e9c-28be-4181-bc8f-987d41d487f2\" (UID: \"d73d0e9c-28be-4181-bc8f-987d41d487f2\") " Nov 03 01:54:56.552354 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.552184 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73d0e9c-28be-4181-bc8f-987d41d487f2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d73d0e9c-28be-4181-bc8f-987d41d487f2-bundle\") pod \"d73d0e9c-28be-4181-bc8f-987d41d487f2\" (UID: \"d73d0e9c-28be-4181-bc8f-987d41d487f2\") " Nov 03 01:54:56.552354 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.552295 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73d0e9c-28be-4181-bc8f-987d41d487f2-util\" (UniqueName: \"kubernetes.io/empty-dir/d73d0e9c-28be-4181-bc8f-987d41d487f2-util\") pod \"d73d0e9c-28be-4181-bc8f-987d41d487f2\" (UID: \"d73d0e9c-28be-4181-bc8f-987d41d487f2\") " Nov 03 01:54:56.552687 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.552666 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73d0e9c-28be-4181-bc8f-987d41d487f2-bundle" pod "d73d0e9c-28be-4181-bc8f-987d41d487f2" (UID: "d73d0e9c-28be-4181-bc8f-987d41d487f2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:56.554246 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.554220 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73d0e9c-28be-4181-bc8f-987d41d487f2-kube-api-access-kfjhb" pod "d73d0e9c-28be-4181-bc8f-987d41d487f2" (UID: "d73d0e9c-28be-4181-bc8f-987d41d487f2"). InnerVolumeSpecName "kube-api-access-kfjhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:54:56.557538 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.557520 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73d0e9c-28be-4181-bc8f-987d41d487f2-util" pod "d73d0e9c-28be-4181-bc8f-987d41d487f2" (UID: "d73d0e9c-28be-4181-bc8f-987d41d487f2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:56.563079 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.563035 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp" podStartSLOduration=3.368767777 podStartE2EDuration="32.563022557s" totalImagesPullingTime="29.19425478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:54:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:54:56.562024581 +0000 UTC m=+178.246133735" watchObservedRunningTime="2026-11-03 01:54:56.563022557 +0000 UTC m=+178.247131712" Nov 03 01:54:56.653684 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.653660 5463 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d73d0e9c-28be-4181-bc8f-987d41d487f2-bundle\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:56.653684 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.653683 5463 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d73d0e9c-28be-4181-bc8f-987d41d487f2-util\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:56.653813 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:56.653692 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfjhb\" (UniqueName: \"kubernetes.io/projected/d73d0e9c-28be-4181-bc8f-987d41d487f2-kube-api-access-kfjhb\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:57.343822 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.343740 5463 generic.go:356] "Generic (PLEG): container finished" podID="14260093-edd8-4d31-abe2-9ce1b3947977" containerID="2c630b7bfa913e53bac6ea11f2721ddc44f14d41332437d69236ea8f2f25b507" exitCode=0 Nov 03 01:54:57.344252 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.343830 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n" event={"ID":"14260093-edd8-4d31-abe2-9ce1b3947977","Type":"ContainerDied","Data":"2c630b7bfa913e53bac6ea11f2721ddc44f14d41332437d69236ea8f2f25b507"} Nov 03 01:54:57.345698 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.345679 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf" event={"ID":"d73d0e9c-28be-4181-bc8f-987d41d487f2","Type":"ContainerDied","Data":"39fe9aaae7d796fc9ed0ed9f731f37c53568d09e4937b041ddca1a1e0200759b"} Nov 03 01:54:57.345698 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.345692 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67jzgf" Nov 03 01:54:57.345820 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.345711 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39fe9aaae7d796fc9ed0ed9f731f37c53568d09e4937b041ddca1a1e0200759b" Nov 03 01:54:57.347883 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.347860 5463 generic.go:356] "Generic (PLEG): container finished" podID="cd0202fb-1f79-404b-b86c-a53d22961e10" containerID="e833c480cb90c21286a0b32576335d6a9c5326cadd63e36f43e8ce0363cdbe1f" exitCode=0 Nov 03 01:54:57.348005 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.347936 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp" event={"ID":"cd0202fb-1f79-404b-b86c-a53d22961e10","Type":"ContainerDied","Data":"e833c480cb90c21286a0b32576335d6a9c5326cadd63e36f43e8ce0363cdbe1f"} Nov 03 01:54:57.470535 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.470513 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv" Nov 03 01:54:57.494556 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.494536 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk" Nov 03 01:54:57.561988 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.561966 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-kube-api-access-8rmbc\" (UniqueName: \"kubernetes.io/projected/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-kube-api-access-8rmbc\") pod \"ea166405-eaf6-4a1d-b09e-7333bc94b8f7\" (UID: \"ea166405-eaf6-4a1d-b09e-7333bc94b8f7\") " Nov 03 01:54:57.562102 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.562011 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-bundle\") pod \"2b7ed792-f8d3-4777-9bd5-58eb728d2a24\" (UID: \"2b7ed792-f8d3-4777-9bd5-58eb728d2a24\") " Nov 03 01:54:57.562102 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.562064 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-bundle\") pod \"ea166405-eaf6-4a1d-b09e-7333bc94b8f7\" (UID: \"ea166405-eaf6-4a1d-b09e-7333bc94b8f7\") " Nov 03 01:54:57.562102 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.562083 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-util\" (UniqueName: \"kubernetes.io/empty-dir/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-util\") pod \"ea166405-eaf6-4a1d-b09e-7333bc94b8f7\" (UID: \"ea166405-eaf6-4a1d-b09e-7333bc94b8f7\") " Nov 03 01:54:57.562273 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.562101 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-kube-api-access-fm2zm\" (UniqueName: \"kubernetes.io/projected/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-kube-api-access-fm2zm\") pod \"2b7ed792-f8d3-4777-9bd5-58eb728d2a24\" (UID: \"2b7ed792-f8d3-4777-9bd5-58eb728d2a24\") " Nov 03 01:54:57.562273 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.562123 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-util\" (UniqueName: \"kubernetes.io/empty-dir/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-util\") pod \"2b7ed792-f8d3-4777-9bd5-58eb728d2a24\" (UID: \"2b7ed792-f8d3-4777-9bd5-58eb728d2a24\") " Nov 03 01:54:57.562556 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.562522 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-bundle" pod "ea166405-eaf6-4a1d-b09e-7333bc94b8f7" (UID: "ea166405-eaf6-4a1d-b09e-7333bc94b8f7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:57.562691 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.562668 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-bundle" pod "2b7ed792-f8d3-4777-9bd5-58eb728d2a24" (UID: "2b7ed792-f8d3-4777-9bd5-58eb728d2a24"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:57.564115 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.564088 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-kube-api-access-8rmbc" pod "ea166405-eaf6-4a1d-b09e-7333bc94b8f7" (UID: "ea166405-eaf6-4a1d-b09e-7333bc94b8f7"). InnerVolumeSpecName "kube-api-access-8rmbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:54:57.564215 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.564122 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-kube-api-access-fm2zm" pod "2b7ed792-f8d3-4777-9bd5-58eb728d2a24" (UID: "2b7ed792-f8d3-4777-9bd5-58eb728d2a24"). InnerVolumeSpecName "kube-api-access-fm2zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:54:57.570739 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.570715 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-util" pod "ea166405-eaf6-4a1d-b09e-7333bc94b8f7" (UID: "ea166405-eaf6-4a1d-b09e-7333bc94b8f7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:57.570950 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.570933 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-util" pod "2b7ed792-f8d3-4777-9bd5-58eb728d2a24" (UID: "2b7ed792-f8d3-4777-9bd5-58eb728d2a24"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:57.663214 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.663178 5463 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-bundle\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:57.663214 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.663214 5463 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-util\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:57.663350 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.663224 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm2zm\" (UniqueName: \"kubernetes.io/projected/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-kube-api-access-fm2zm\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:57.663350 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.663235 5463 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-util\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:57.663350 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.663243 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rmbc\" (UniqueName: \"kubernetes.io/projected/ea166405-eaf6-4a1d-b09e-7333bc94b8f7-kube-api-access-8rmbc\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:57.663350 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:57.663252 5463 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b7ed792-f8d3-4777-9bd5-58eb728d2a24-bundle\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:58.354204 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.354158 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv" event={"ID":"ea166405-eaf6-4a1d-b09e-7333bc94b8f7","Type":"ContainerDied","Data":"f1b928dc6327226b3a24df31953ec75295ae603926c2299f72a08e19d22d4cd9"} Nov 03 01:54:58.354204 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.354176 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nfmnv" Nov 03 01:54:58.354677 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.354229 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1b928dc6327226b3a24df31953ec75295ae603926c2299f72a08e19d22d4cd9" Nov 03 01:54:58.356066 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.356040 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk" event={"ID":"2b7ed792-f8d3-4777-9bd5-58eb728d2a24","Type":"ContainerDied","Data":"ee9784888bc4626d3aa48065f085ae306b3eae655239aaa95676110b4678c1c0"} Nov 03 01:54:58.356183 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.356067 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpxsfk" Nov 03 01:54:58.356183 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.356084 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee9784888bc4626d3aa48065f085ae306b3eae655239aaa95676110b4678c1c0" Nov 03 01:54:58.497960 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.497938 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp" Nov 03 01:54:58.499941 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.499923 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n" Nov 03 01:54:58.582769 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.582747 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0202fb-1f79-404b-b86c-a53d22961e10-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd0202fb-1f79-404b-b86c-a53d22961e10-bundle\") pod \"cd0202fb-1f79-404b-b86c-a53d22961e10\" (UID: \"cd0202fb-1f79-404b-b86c-a53d22961e10\") " Nov 03 01:54:58.582896 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.582798 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0202fb-1f79-404b-b86c-a53d22961e10-kube-api-access-cw9t4\" (UniqueName: \"kubernetes.io/projected/cd0202fb-1f79-404b-b86c-a53d22961e10-kube-api-access-cw9t4\") pod \"cd0202fb-1f79-404b-b86c-a53d22961e10\" (UID: \"cd0202fb-1f79-404b-b86c-a53d22961e10\") " Nov 03 01:54:58.582896 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.582838 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14260093-edd8-4d31-abe2-9ce1b3947977-bundle\" (UniqueName: \"kubernetes.io/empty-dir/14260093-edd8-4d31-abe2-9ce1b3947977-bundle\") pod \"14260093-edd8-4d31-abe2-9ce1b3947977\" (UID: \"14260093-edd8-4d31-abe2-9ce1b3947977\") " Nov 03 01:54:58.582896 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.582881 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0202fb-1f79-404b-b86c-a53d22961e10-util\" (UniqueName: \"kubernetes.io/empty-dir/cd0202fb-1f79-404b-b86c-a53d22961e10-util\") pod \"cd0202fb-1f79-404b-b86c-a53d22961e10\" (UID: \"cd0202fb-1f79-404b-b86c-a53d22961e10\") " Nov 03 01:54:58.583019 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.582901 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14260093-edd8-4d31-abe2-9ce1b3947977-kube-api-access-qkgdp\" (UniqueName: \"kubernetes.io/projected/14260093-edd8-4d31-abe2-9ce1b3947977-kube-api-access-qkgdp\") pod \"14260093-edd8-4d31-abe2-9ce1b3947977\" (UID: \"14260093-edd8-4d31-abe2-9ce1b3947977\") " Nov 03 01:54:58.583058 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.583027 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14260093-edd8-4d31-abe2-9ce1b3947977-util\" (UniqueName: \"kubernetes.io/empty-dir/14260093-edd8-4d31-abe2-9ce1b3947977-util\") pod \"14260093-edd8-4d31-abe2-9ce1b3947977\" (UID: \"14260093-edd8-4d31-abe2-9ce1b3947977\") " Nov 03 01:54:58.583570 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.583538 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0202fb-1f79-404b-b86c-a53d22961e10-bundle" pod "cd0202fb-1f79-404b-b86c-a53d22961e10" (UID: "cd0202fb-1f79-404b-b86c-a53d22961e10"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:58.583697 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.583642 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14260093-edd8-4d31-abe2-9ce1b3947977-bundle" pod "14260093-edd8-4d31-abe2-9ce1b3947977" (UID: "14260093-edd8-4d31-abe2-9ce1b3947977"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:58.585106 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.585077 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0202fb-1f79-404b-b86c-a53d22961e10-kube-api-access-cw9t4" pod "cd0202fb-1f79-404b-b86c-a53d22961e10" (UID: "cd0202fb-1f79-404b-b86c-a53d22961e10"). InnerVolumeSpecName "kube-api-access-cw9t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:54:58.585600 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.585576 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14260093-edd8-4d31-abe2-9ce1b3947977-kube-api-access-qkgdp" pod "14260093-edd8-4d31-abe2-9ce1b3947977" (UID: "14260093-edd8-4d31-abe2-9ce1b3947977"). InnerVolumeSpecName "kube-api-access-qkgdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:54:58.589219 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.589076 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0202fb-1f79-404b-b86c-a53d22961e10-util" pod "cd0202fb-1f79-404b-b86c-a53d22961e10" (UID: "cd0202fb-1f79-404b-b86c-a53d22961e10"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:58.590328 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.590305 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14260093-edd8-4d31-abe2-9ce1b3947977-util" pod "14260093-edd8-4d31-abe2-9ce1b3947977" (UID: "14260093-edd8-4d31-abe2-9ce1b3947977"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:58.684474 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.684446 5463 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd0202fb-1f79-404b-b86c-a53d22961e10-util\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:58.684474 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.684471 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkgdp\" (UniqueName: \"kubernetes.io/projected/14260093-edd8-4d31-abe2-9ce1b3947977-kube-api-access-qkgdp\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:58.684660 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.684483 5463 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14260093-edd8-4d31-abe2-9ce1b3947977-util\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:58.684660 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.684492 5463 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd0202fb-1f79-404b-b86c-a53d22961e10-bundle\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:58.684660 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.684499 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw9t4\" (UniqueName: \"kubernetes.io/projected/cd0202fb-1f79-404b-b86c-a53d22961e10-kube-api-access-cw9t4\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:58.684660 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:58.684507 5463 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14260093-edd8-4d31-abe2-9ce1b3947977-bundle\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:59.361737 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:59.361712 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n" Nov 03 01:54:59.362170 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:59.361706 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj6d5n" event={"ID":"14260093-edd8-4d31-abe2-9ce1b3947977","Type":"ContainerDied","Data":"42bf727a131a489ab12f65318c52b6ddf134320d08ccbf39b64c59306809941b"} Nov 03 01:54:59.362170 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:59.361879 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42bf727a131a489ab12f65318c52b6ddf134320d08ccbf39b64c59306809941b" Nov 03 01:54:59.363677 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:59.363650 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp" event={"ID":"cd0202fb-1f79-404b-b86c-a53d22961e10","Type":"ContainerDied","Data":"c69c79a802e89b86797aaec31d1c1ccd865d6a2af7ee0b71398d5e3c708a8c11"} Nov 03 01:54:59.363792 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:59.363691 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c69c79a802e89b86797aaec31d1c1ccd865d6a2af7ee0b71398d5e3c708a8c11" Nov 03 01:54:59.363792 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:59.363693 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lfrfp" Nov 03 01:54:59.365456 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:54:59.365431 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:00.173783 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:00.173752 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:02.973881 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.973849 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hvn6t"] Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974159 5463 cpu_manager.go:414] "containerMap: removing container" podUID="ea166405-eaf6-4a1d-b09e-7333bc94b8f7" containerName="pull" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974170 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea166405-eaf6-4a1d-b09e-7333bc94b8f7" containerName="pull" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974183 5463 cpu_manager.go:414] "containerMap: removing container" podUID="14260093-edd8-4d31-abe2-9ce1b3947977" containerName="pull" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974201 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14260093-edd8-4d31-abe2-9ce1b3947977" containerName="pull" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974208 5463 cpu_manager.go:414] "containerMap: removing container" podUID="14260093-edd8-4d31-abe2-9ce1b3947977" containerName="extract" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974212 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14260093-edd8-4d31-abe2-9ce1b3947977" containerName="extract" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974221 5463 cpu_manager.go:414] "containerMap: removing container" podUID="cd0202fb-1f79-404b-b86c-a53d22961e10" containerName="pull" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974225 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0202fb-1f79-404b-b86c-a53d22961e10" containerName="pull" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974232 5463 cpu_manager.go:414] "containerMap: removing container" podUID="ea166405-eaf6-4a1d-b09e-7333bc94b8f7" containerName="extract" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974236 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea166405-eaf6-4a1d-b09e-7333bc94b8f7" containerName="extract" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974245 5463 cpu_manager.go:414] "containerMap: removing container" podUID="bb234da4-434c-4c86-9bbd-7f69281a5945" containerName="util" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974249 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb234da4-434c-4c86-9bbd-7f69281a5945" containerName="util" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974258 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d73d0e9c-28be-4181-bc8f-987d41d487f2" containerName="util" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974262 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73d0e9c-28be-4181-bc8f-987d41d487f2" containerName="util" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974270 5463 cpu_manager.go:414] "containerMap: removing container" podUID="c1f4153b-edb6-4e8d-93c5-c14eaa71e741" containerName="extract" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974276 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f4153b-edb6-4e8d-93c5-c14eaa71e741" containerName="extract" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974282 5463 cpu_manager.go:414] "containerMap: removing container" podUID="cd0202fb-1f79-404b-b86c-a53d22961e10" containerName="extract" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974287 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0202fb-1f79-404b-b86c-a53d22961e10" containerName="extract" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974293 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2b7ed792-f8d3-4777-9bd5-58eb728d2a24" containerName="extract" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974297 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b7ed792-f8d3-4777-9bd5-58eb728d2a24" containerName="extract" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974305 5463 cpu_manager.go:414] "containerMap: removing container" podUID="14260093-edd8-4d31-abe2-9ce1b3947977" containerName="util" Nov 03 01:55:02.974296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974309 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14260093-edd8-4d31-abe2-9ce1b3947977" containerName="util" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974317 5463 cpu_manager.go:414] "containerMap: removing container" podUID="ea166405-eaf6-4a1d-b09e-7333bc94b8f7" containerName="util" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974321 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea166405-eaf6-4a1d-b09e-7333bc94b8f7" containerName="util" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974327 5463 cpu_manager.go:414] "containerMap: removing container" podUID="bb234da4-434c-4c86-9bbd-7f69281a5945" containerName="pull" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974331 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb234da4-434c-4c86-9bbd-7f69281a5945" containerName="pull" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974337 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2b7ed792-f8d3-4777-9bd5-58eb728d2a24" containerName="pull" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974341 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b7ed792-f8d3-4777-9bd5-58eb728d2a24" containerName="pull" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974348 5463 cpu_manager.go:414] "containerMap: removing container" podUID="c1f4153b-edb6-4e8d-93c5-c14eaa71e741" containerName="util" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974353 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f4153b-edb6-4e8d-93c5-c14eaa71e741" containerName="util" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974361 5463 cpu_manager.go:414] "containerMap: removing container" podUID="c1f4153b-edb6-4e8d-93c5-c14eaa71e741" containerName="pull" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974365 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f4153b-edb6-4e8d-93c5-c14eaa71e741" containerName="pull" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974371 5463 cpu_manager.go:414] "containerMap: removing container" podUID="bb234da4-434c-4c86-9bbd-7f69281a5945" containerName="extract" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974375 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb234da4-434c-4c86-9bbd-7f69281a5945" containerName="extract" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974381 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d73d0e9c-28be-4181-bc8f-987d41d487f2" containerName="extract" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974385 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73d0e9c-28be-4181-bc8f-987d41d487f2" containerName="extract" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974392 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d73d0e9c-28be-4181-bc8f-987d41d487f2" containerName="pull" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974396 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73d0e9c-28be-4181-bc8f-987d41d487f2" containerName="pull" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974403 5463 cpu_manager.go:414] "containerMap: removing container" podUID="cd0202fb-1f79-404b-b86c-a53d22961e10" containerName="util" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974407 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0202fb-1f79-404b-b86c-a53d22961e10" containerName="util" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974414 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2b7ed792-f8d3-4777-9bd5-58eb728d2a24" containerName="util" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974418 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b7ed792-f8d3-4777-9bd5-58eb728d2a24" containerName="util" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974460 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73d0e9c-28be-4181-bc8f-987d41d487f2" containerName="extract" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974466 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb234da4-434c-4c86-9bbd-7f69281a5945" containerName="extract" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974475 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="14260093-edd8-4d31-abe2-9ce1b3947977" containerName="extract" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974480 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea166405-eaf6-4a1d-b09e-7333bc94b8f7" containerName="extract" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974486 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f4153b-edb6-4e8d-93c5-c14eaa71e741" containerName="extract" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974490 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0202fb-1f79-404b-b86c-a53d22961e10" containerName="extract" Nov 03 01:55:02.974937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.974495 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b7ed792-f8d3-4777-9bd5-58eb728d2a24" containerName="extract" Nov 03 01:55:02.977780 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:02.977756 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hvn6t" Nov 03 01:55:03.034118 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:03.034087 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 01:55:03.034281 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:03.034152 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 01:55:03.034281 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:03.034158 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-46trd\"" Nov 03 01:55:03.068621 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:03.068592 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hvn6t"] Nov 03 01:55:03.116082 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:03.116059 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xq7b\" (UniqueName: \"kubernetes.io/projected/978d7bd0-c834-455c-90fc-8388993542d6-kube-api-access-2xq7b\") pod \"authorino-operator-7b7bb4bc77-hvn6t\" (UID: \"978d7bd0-c834-455c-90fc-8388993542d6\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hvn6t" Nov 03 01:55:03.216789 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:03.216762 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xq7b\" (UniqueName: \"kubernetes.io/projected/978d7bd0-c834-455c-90fc-8388993542d6-kube-api-access-2xq7b\") pod \"authorino-operator-7b7bb4bc77-hvn6t\" (UID: \"978d7bd0-c834-455c-90fc-8388993542d6\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hvn6t" Nov 03 01:55:03.276494 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:03.276445 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xq7b\" (UniqueName: \"kubernetes.io/projected/978d7bd0-c834-455c-90fc-8388993542d6-kube-api-access-2xq7b\") pod \"authorino-operator-7b7bb4bc77-hvn6t\" (UID: \"978d7bd0-c834-455c-90fc-8388993542d6\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hvn6t" Nov 03 01:55:03.289215 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:03.289179 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hvn6t" Nov 03 01:55:03.655665 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:03.655635 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hvn6t"] Nov 03 01:55:03.687034 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:55:03.687002 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978d7bd0_c834_455c_90fc_8388993542d6.slice/crio-c72d602b4d1eb55f6ff6b893ab5a02b63e39c3b83c5cddd0e82686e995f8c8b8 WatchSource:0}: Error finding container c72d602b4d1eb55f6ff6b893ab5a02b63e39c3b83c5cddd0e82686e995f8c8b8: Status 404 returned error can't find the container with id c72d602b4d1eb55f6ff6b893ab5a02b63e39c3b83c5cddd0e82686e995f8c8b8 Nov 03 01:55:03.778222 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:03.778175 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:04.176144 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:04.176117 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:04.385836 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:04.385797 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hvn6t" event={"ID":"978d7bd0-c834-455c-90fc-8388993542d6","Type":"ContainerStarted","Data":"c72d602b4d1eb55f6ff6b893ab5a02b63e39c3b83c5cddd0e82686e995f8c8b8"} Nov 03 01:55:04.537801 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:04.537706 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:04.823476 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:04.823396 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:04.995841 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:04.995807 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:09.089406 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:09.089308 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:09.411970 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:09.411939 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hvn6t" event={"ID":"978d7bd0-c834-455c-90fc-8388993542d6","Type":"ContainerStarted","Data":"14c2ee472633794274b20cffabc1d6bb588e9f72156e6c79759e0e5b9e326e0f"} Nov 03 01:55:09.412133 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:09.412116 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hvn6t" Nov 03 01:55:09.527002 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:09.526952 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hvn6t" podStartSLOduration=2.472030276 podStartE2EDuration="7.526939947s" totalImagesPullingTime="5.054909671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:09.526075112 +0000 UTC m=+191.210184250" watchObservedRunningTime="2026-11-03 01:55:09.526939947 +0000 UTC m=+191.211049099" Nov 03 01:55:10.177736 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:10.177694 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:10.352427 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:10.352393 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:10.672912 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:10.672882 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb"] Nov 03 01:55:10.675070 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:10.675050 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb" Nov 03 01:55:10.716846 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:10.716820 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb"] Nov 03 01:55:10.765926 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:10.765906 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-c5vrr\"" Nov 03 01:55:10.779886 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:10.779854 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7861e771-80dd-4382-919a-ef02cb7d0892-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-dhsqb\" (UID: \"7861e771-80dd-4382-919a-ef02cb7d0892\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb" Nov 03 01:55:10.780000 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:10.779901 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2t9s\" (UniqueName: \"kubernetes.io/projected/7861e771-80dd-4382-919a-ef02cb7d0892-kube-api-access-x2t9s\") pod \"kuadrant-operator-controller-manager-cf8469d7b-dhsqb\" (UID: \"7861e771-80dd-4382-919a-ef02cb7d0892\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb" Nov 03 01:55:10.880814 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:10.880784 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2t9s\" (UniqueName: \"kubernetes.io/projected/7861e771-80dd-4382-919a-ef02cb7d0892-kube-api-access-x2t9s\") pod \"kuadrant-operator-controller-manager-cf8469d7b-dhsqb\" (UID: \"7861e771-80dd-4382-919a-ef02cb7d0892\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb" Nov 03 01:55:10.881018 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:10.880863 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7861e771-80dd-4382-919a-ef02cb7d0892-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-dhsqb\" (UID: \"7861e771-80dd-4382-919a-ef02cb7d0892\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb" Nov 03 01:55:10.881282 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:10.881262 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7861e771-80dd-4382-919a-ef02cb7d0892-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-dhsqb\" (UID: \"7861e771-80dd-4382-919a-ef02cb7d0892\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb" Nov 03 01:55:10.955949 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:10.955887 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2t9s\" (UniqueName: \"kubernetes.io/projected/7861e771-80dd-4382-919a-ef02cb7d0892-kube-api-access-x2t9s\") pod \"kuadrant-operator-controller-manager-cf8469d7b-dhsqb\" (UID: \"7861e771-80dd-4382-919a-ef02cb7d0892\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb" Nov 03 01:55:10.985831 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:10.985798 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb" Nov 03 01:55:11.448609 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.448580 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb"] Nov 03 01:55:11.479069 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:55:11.479040 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7861e771_80dd_4382_919a_ef02cb7d0892.slice/crio-ac59c197040384f3d5e846fadbd20128c9a5a6dc7c020dd0d1ad991a32c6d551 WatchSource:0}: Error finding container ac59c197040384f3d5e846fadbd20128c9a5a6dc7c020dd0d1ad991a32c6d551: Status 404 returned error can't find the container with id ac59c197040384f3d5e846fadbd20128c9a5a6dc7c020dd0d1ad991a32c6d551 Nov 03 01:55:11.512747 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.512722 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:11.699550 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.699477 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh"] Nov 03 01:55:11.702699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.702682 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:11.729667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.729646 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh"] Nov 03 01:55:11.755530 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.755502 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 01:55:11.781561 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.781515 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hwpp2\"" Nov 03 01:55:11.781561 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.781550 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 01:55:11.781704 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.781555 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 01:55:11.781704 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.781610 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 01:55:11.787630 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.787609 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfdkk\" (UniqueName: \"kubernetes.io/projected/3b225bf7-3800-423e-8334-8d6afae3f5ce-kube-api-access-qfdkk\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-dxphh\" (UID: \"3b225bf7-3800-423e-8334-8d6afae3f5ce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:11.787700 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.787640 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b225bf7-3800-423e-8334-8d6afae3f5ce-metrics-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-dxphh\" (UID: \"3b225bf7-3800-423e-8334-8d6afae3f5ce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:11.787700 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.787661 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3b225bf7-3800-423e-8334-8d6afae3f5ce-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-dxphh\" (UID: \"3b225bf7-3800-423e-8334-8d6afae3f5ce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:11.787788 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.787765 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b225bf7-3800-423e-8334-8d6afae3f5ce-tmp\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-dxphh\" (UID: \"3b225bf7-3800-423e-8334-8d6afae3f5ce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:11.787866 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.787846 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3b225bf7-3800-423e-8334-8d6afae3f5ce-manifests\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-dxphh\" (UID: \"3b225bf7-3800-423e-8334-8d6afae3f5ce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:11.787937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.787925 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b225bf7-3800-423e-8334-8d6afae3f5ce-webhook-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-dxphh\" (UID: \"3b225bf7-3800-423e-8334-8d6afae3f5ce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:11.826160 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.826130 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:11.855459 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.855438 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 01:55:11.889336 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.889295 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b225bf7-3800-423e-8334-8d6afae3f5ce-webhook-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-dxphh\" (UID: \"3b225bf7-3800-423e-8334-8d6afae3f5ce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:11.889443 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.889368 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdkk\" (UniqueName: \"kubernetes.io/projected/3b225bf7-3800-423e-8334-8d6afae3f5ce-kube-api-access-qfdkk\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-dxphh\" (UID: \"3b225bf7-3800-423e-8334-8d6afae3f5ce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:11.889443 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.889399 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b225bf7-3800-423e-8334-8d6afae3f5ce-metrics-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-dxphh\" (UID: \"3b225bf7-3800-423e-8334-8d6afae3f5ce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:11.889443 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.889426 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3b225bf7-3800-423e-8334-8d6afae3f5ce-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-dxphh\" (UID: \"3b225bf7-3800-423e-8334-8d6afae3f5ce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:11.889592 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.889458 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b225bf7-3800-423e-8334-8d6afae3f5ce-tmp\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-dxphh\" (UID: \"3b225bf7-3800-423e-8334-8d6afae3f5ce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:11.889592 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.889492 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3b225bf7-3800-423e-8334-8d6afae3f5ce-manifests\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-dxphh\" (UID: \"3b225bf7-3800-423e-8334-8d6afae3f5ce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:11.889901 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.889878 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3b225bf7-3800-423e-8334-8d6afae3f5ce-manifests\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-dxphh\" (UID: \"3b225bf7-3800-423e-8334-8d6afae3f5ce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:11.892016 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.891991 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b225bf7-3800-423e-8334-8d6afae3f5ce-metrics-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-dxphh\" (UID: \"3b225bf7-3800-423e-8334-8d6afae3f5ce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:11.892133 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.892041 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b225bf7-3800-423e-8334-8d6afae3f5ce-tmp\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-dxphh\" (UID: \"3b225bf7-3800-423e-8334-8d6afae3f5ce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:11.892255 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.892237 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3b225bf7-3800-423e-8334-8d6afae3f5ce-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-dxphh\" (UID: \"3b225bf7-3800-423e-8334-8d6afae3f5ce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:11.892327 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.892272 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b225bf7-3800-423e-8334-8d6afae3f5ce-webhook-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-dxphh\" (UID: \"3b225bf7-3800-423e-8334-8d6afae3f5ce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:11.954619 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.954557 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfdkk\" (UniqueName: \"kubernetes.io/projected/3b225bf7-3800-423e-8334-8d6afae3f5ce-kube-api-access-qfdkk\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-dxphh\" (UID: \"3b225bf7-3800-423e-8334-8d6afae3f5ce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:11.993687 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:11.993655 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:12.016754 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:12.016731 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:12.158701 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:12.158666 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:12.332502 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:12.332436 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:12.425044 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:12.425015 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb" event={"ID":"7861e771-80dd-4382-919a-ef02cb7d0892","Type":"ContainerStarted","Data":"ac59c197040384f3d5e846fadbd20128c9a5a6dc7c020dd0d1ad991a32c6d551"} Nov 03 01:55:12.483616 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:55:12.483588 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b225bf7_3800_423e_8334_8d6afae3f5ce.slice/crio-69092699da94d3ec7f59a7bee9d81e04f7a1d06e8b9c2ff02bcb838de458286e WatchSource:0}: Error finding container 69092699da94d3ec7f59a7bee9d81e04f7a1d06e8b9c2ff02bcb838de458286e: Status 404 returned error can't find the container with id 69092699da94d3ec7f59a7bee9d81e04f7a1d06e8b9c2ff02bcb838de458286e Nov 03 01:55:12.485295 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:12.485051 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh"] Nov 03 01:55:13.004321 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:55:13.004284 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 01:55:13.215433 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:13.215401 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:13.430233 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:13.430199 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" event={"ID":"3b225bf7-3800-423e-8334-8d6afae3f5ce","Type":"ContainerStarted","Data":"69092699da94d3ec7f59a7bee9d81e04f7a1d06e8b9c2ff02bcb838de458286e"} Nov 03 01:55:14.813775 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:14.813744 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-h8jgm"] Nov 03 01:55:14.816290 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:14.816267 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h8jgm" Nov 03 01:55:14.872252 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:14.872218 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-h8jgm"] Nov 03 01:55:14.917175 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:14.917135 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhcpw\" (UniqueName: \"kubernetes.io/projected/2d19ed0a-2c09-4b4c-a4bd-cb1f64032fc5-kube-api-access-lhcpw\") pod \"limitador-operator-controller-manager-5778bb4449-h8jgm\" (UID: \"2d19ed0a-2c09-4b4c-a4bd-cb1f64032fc5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h8jgm" Nov 03 01:55:14.921752 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:14.921692 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-r64fn\"" Nov 03 01:55:15.018505 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:15.018464 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhcpw\" (UniqueName: \"kubernetes.io/projected/2d19ed0a-2c09-4b4c-a4bd-cb1f64032fc5-kube-api-access-lhcpw\") pod \"limitador-operator-controller-manager-5778bb4449-h8jgm\" (UID: \"2d19ed0a-2c09-4b4c-a4bd-cb1f64032fc5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h8jgm" Nov 03 01:55:15.027242 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:15.027207 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:15.089943 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:15.089882 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhcpw\" (UniqueName: \"kubernetes.io/projected/2d19ed0a-2c09-4b4c-a4bd-cb1f64032fc5-kube-api-access-lhcpw\") pod \"limitador-operator-controller-manager-5778bb4449-h8jgm\" (UID: \"2d19ed0a-2c09-4b4c-a4bd-cb1f64032fc5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h8jgm" Nov 03 01:55:15.130261 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:15.130224 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h8jgm" Nov 03 01:55:15.195570 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:15.195535 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:15.512866 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:15.512834 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-h8jgm"] Nov 03 01:55:15.543404 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:55:15.543375 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d19ed0a_2c09_4b4c_a4bd_cb1f64032fc5.slice/crio-4edbe8018ddcf7cd7110117af96b33a27d9165725855954852e609e1d7554e71 WatchSource:0}: Error finding container 4edbe8018ddcf7cd7110117af96b33a27d9165725855954852e609e1d7554e71: Status 404 returned error can't find the container with id 4edbe8018ddcf7cd7110117af96b33a27d9165725855954852e609e1d7554e71 Nov 03 01:55:16.445623 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:16.445546 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:16.449575 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:16.449543 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h8jgm" event={"ID":"2d19ed0a-2c09-4b4c-a4bd-cb1f64032fc5","Type":"ContainerStarted","Data":"4edbe8018ddcf7cd7110117af96b33a27d9165725855954852e609e1d7554e71"} Nov 03 01:55:17.455372 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:17.455337 5463 generic.go:356] "Generic (PLEG): container finished" podID="3b225bf7-3800-423e-8334-8d6afae3f5ce" containerID="cbc698d939e5598f730dae519e216add9e1a3083c594fa6a437e4271d7a5d1b0" exitCode=0 Nov 03 01:55:17.455717 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:17.455404 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" event={"ID":"3b225bf7-3800-423e-8334-8d6afae3f5ce","Type":"ContainerDied","Data":"cbc698d939e5598f730dae519e216add9e1a3083c594fa6a437e4271d7a5d1b0"} Nov 03 01:55:17.850690 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:17.850663 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:18.460983 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:18.460944 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" event={"ID":"3b225bf7-3800-423e-8334-8d6afae3f5ce","Type":"ContainerStarted","Data":"2d1cb7e73e5c9a3876167eba570976f105e87a67949cd99e0be7128dce3f2090"} Nov 03 01:55:18.461490 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:18.461029 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:18.691988 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:18.691930 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" podStartSLOduration=2.147989152 podStartE2EDuration="7.691917008s" totalImagesPullingTime="5.044986579s" totalInitContainerRuntime="498.941277ms" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:18.641149654 +0000 UTC m=+200.325258809" watchObservedRunningTime="2026-11-03 01:55:18.691917008 +0000 UTC m=+200.376026162" Nov 03 01:55:19.528576 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:19.528541 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-qczcb"] Nov 03 01:55:19.532314 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:19.532293 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qczcb" Nov 03 01:55:19.559549 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:19.559514 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxx9s\" (UniqueName: \"kubernetes.io/projected/53d43c23-ca28-4ff5-874c-48e53b872e0c-kube-api-access-xxx9s\") pod \"dns-operator-controller-manager-8557cbdb84-qczcb\" (UID: \"53d43c23-ca28-4ff5-874c-48e53b872e0c\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qczcb" Nov 03 01:55:19.585879 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:19.585846 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-qczcb"] Nov 03 01:55:19.645095 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:19.645064 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-qg25h\"" Nov 03 01:55:19.651640 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:19.651615 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 01:55:19.660867 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:19.660830 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxx9s\" (UniqueName: \"kubernetes.io/projected/53d43c23-ca28-4ff5-874c-48e53b872e0c-kube-api-access-xxx9s\") pod \"dns-operator-controller-manager-8557cbdb84-qczcb\" (UID: \"53d43c23-ca28-4ff5-874c-48e53b872e0c\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qczcb" Nov 03 01:55:19.720543 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:19.720336 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxx9s\" (UniqueName: \"kubernetes.io/projected/53d43c23-ca28-4ff5-874c-48e53b872e0c-kube-api-access-xxx9s\") pod \"dns-operator-controller-manager-8557cbdb84-qczcb\" (UID: \"53d43c23-ca28-4ff5-874c-48e53b872e0c\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qczcb" Nov 03 01:55:19.785641 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:19.785545 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:19.849458 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:19.849420 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qczcb" Nov 03 01:55:20.207436 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:20.207405 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:20.231742 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:55:20.231703 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d43c23_ca28_4ff5_874c_48e53b872e0c.slice/crio-3ff390b857be4e9d170e3096e76365a0746288f881b4d66804ca232ae8f2b691 WatchSource:0}: Error finding container 3ff390b857be4e9d170e3096e76365a0746288f881b4d66804ca232ae8f2b691: Status 404 returned error can't find the container with id 3ff390b857be4e9d170e3096e76365a0746288f881b4d66804ca232ae8f2b691 Nov 03 01:55:20.233071 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:20.233049 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-qczcb"] Nov 03 01:55:20.418804 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:20.418774 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hvn6t" Nov 03 01:55:20.473933 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:20.473852 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qczcb" event={"ID":"53d43c23-ca28-4ff5-874c-48e53b872e0c","Type":"ContainerStarted","Data":"3ff390b857be4e9d170e3096e76365a0746288f881b4d66804ca232ae8f2b691"} Nov 03 01:55:21.479748 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:21.479674 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h8jgm" event={"ID":"2d19ed0a-2c09-4b4c-a4bd-cb1f64032fc5","Type":"ContainerStarted","Data":"da4102b365e374a0f1d5186cc738436f3f200e637b40150cf27ddde56ee616ce"} Nov 03 01:55:21.480090 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:21.479821 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h8jgm" Nov 03 01:55:21.595467 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:21.595419 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h8jgm" podStartSLOduration=1.912294613 podStartE2EDuration="7.595402678s" totalImagesPullingTime="5.683108065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:21.594304309 +0000 UTC m=+203.278413466" watchObservedRunningTime="2026-11-03 01:55:21.595402678 +0000 UTC m=+203.279511831" Nov 03 01:55:21.807010 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:21.806909 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:21.986516 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:21.986472 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:22.143667 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:22.143628 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:22.970456 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:22.970400 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:23.123392 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:23.123365 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:26.506676 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:26.506573 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb" event={"ID":"7861e771-80dd-4382-919a-ef02cb7d0892","Type":"ContainerStarted","Data":"9a27a092a5ebcb9ca39a28308f0ed007ea365f317b5d5a620c1eef7ccb16d99d"} Nov 03 01:55:26.507139 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:26.506688 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb" Nov 03 01:55:26.508152 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:26.508125 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qczcb" event={"ID":"53d43c23-ca28-4ff5-874c-48e53b872e0c","Type":"ContainerStarted","Data":"444375c0af018cc0461c8885928b9b8f29f2be5dc49e6cef140906bd27686afa"} Nov 03 01:55:26.508283 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:26.508249 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qczcb" Nov 03 01:55:26.729052 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:26.728994 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb" podStartSLOduration=2.655927561 podStartE2EDuration="16.728980309s" totalImagesPullingTime="14.073052748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:26.677595713 +0000 UTC m=+208.361704870" watchObservedRunningTime="2026-11-03 01:55:26.728980309 +0000 UTC m=+208.413089463" Nov 03 01:55:26.794341 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:26.794243 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qczcb" podStartSLOduration=1.816589482 podStartE2EDuration="7.79422527s" totalImagesPullingTime="5.977635788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:26.793084629 +0000 UTC m=+208.477193785" watchObservedRunningTime="2026-11-03 01:55:26.79422527 +0000 UTC m=+208.478334426" Nov 03 01:55:28.958714 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:28.958671 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:29.102382 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:29.102345 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:29.468290 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:29.468259 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-dxphh" Nov 03 01:55:29.574562 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:29.574530 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:29.735159 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:29.735087 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:30.197933 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:30.197906 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:30.429036 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:30.429004 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:30.871507 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:30.871472 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:31.043858 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:31.043824 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:31.193638 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:31.193608 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:32.488153 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:32.488119 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h8jgm" Nov 03 01:55:33.459204 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:33.459161 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:35.705400 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:35.705366 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:37.514983 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:37.514944 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb" Nov 03 01:55:37.515435 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:37.515175 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qczcb" Nov 03 01:55:37.772073 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:37.771988 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:37.939135 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:37.939085 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:38.014523 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:38.014489 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt"] Nov 03 01:55:38.017462 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:38.017444 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt" Nov 03 01:55:38.073855 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:38.073828 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt"] Nov 03 01:55:38.121083 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:38.121049 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a759ef00-49b6-406f-8a47-255bd11ca523-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-sgplt\" (UID: \"a759ef00-49b6-406f-8a47-255bd11ca523\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt" Nov 03 01:55:38.121254 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:38.121160 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjn78\" (UniqueName: \"kubernetes.io/projected/a759ef00-49b6-406f-8a47-255bd11ca523-kube-api-access-fjn78\") pod \"kuadrant-operator-controller-manager-678d87766f-sgplt\" (UID: \"a759ef00-49b6-406f-8a47-255bd11ca523\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt" Nov 03 01:55:38.222357 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:38.222322 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a759ef00-49b6-406f-8a47-255bd11ca523-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-sgplt\" (UID: \"a759ef00-49b6-406f-8a47-255bd11ca523\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt" Nov 03 01:55:38.222523 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:38.222415 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjn78\" (UniqueName: \"kubernetes.io/projected/a759ef00-49b6-406f-8a47-255bd11ca523-kube-api-access-fjn78\") pod \"kuadrant-operator-controller-manager-678d87766f-sgplt\" (UID: \"a759ef00-49b6-406f-8a47-255bd11ca523\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt" Nov 03 01:55:38.222710 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:38.222690 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a759ef00-49b6-406f-8a47-255bd11ca523-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-sgplt\" (UID: \"a759ef00-49b6-406f-8a47-255bd11ca523\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt" Nov 03 01:55:38.282992 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:38.282925 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjn78\" (UniqueName: \"kubernetes.io/projected/a759ef00-49b6-406f-8a47-255bd11ca523-kube-api-access-fjn78\") pod \"kuadrant-operator-controller-manager-678d87766f-sgplt\" (UID: \"a759ef00-49b6-406f-8a47-255bd11ca523\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt" Nov 03 01:55:38.328972 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:38.328948 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt" Nov 03 01:55:38.789405 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:55:38.789374 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda759ef00_49b6_406f_8a47_255bd11ca523.slice/crio-594955f5eee04191aec936ac6e83092a87e2fbbc0ee2f2a87f267c52567f57c5 WatchSource:0}: Error finding container 594955f5eee04191aec936ac6e83092a87e2fbbc0ee2f2a87f267c52567f57c5: Status 404 returned error can't find the container with id 594955f5eee04191aec936ac6e83092a87e2fbbc0ee2f2a87f267c52567f57c5 Nov 03 01:55:38.791174 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:38.791149 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt"] Nov 03 01:55:39.569345 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:39.569309 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt" event={"ID":"a759ef00-49b6-406f-8a47-255bd11ca523","Type":"ContainerStarted","Data":"450c99b991652262c75bfc18c75941504a830ba5c34c09db43edaac7e3de4c39"} Nov 03 01:55:39.569545 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:39.569365 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt" event={"ID":"a759ef00-49b6-406f-8a47-255bd11ca523","Type":"ContainerStarted","Data":"594955f5eee04191aec936ac6e83092a87e2fbbc0ee2f2a87f267c52567f57c5"} Nov 03 01:55:39.569545 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:39.569466 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt" Nov 03 01:55:39.790496 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:39.790445 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt" podStartSLOduration=2.7904310150000002 podStartE2EDuration="2.790431015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:39.739398893 +0000 UTC m=+221.423508061" watchObservedRunningTime="2026-11-03 01:55:39.790431015 +0000 UTC m=+221.474540169" Nov 03 01:55:40.770965 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:40.770930 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:40.920284 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:40.920253 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:41.421657 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:41.421625 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:42.689625 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:42.689594 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:42.852905 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:42.852873 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:43.005690 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:55:43.005611 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 01:55:43.196220 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:43.196160 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:44.015447 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:44.015414 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:44.168765 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:44.168736 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:44.418456 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:44.418421 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:44.669795 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:44.669715 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:44.827559 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:44.827521 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:45.174002 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.173972 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:45.329545 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.329508 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:45.395580 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.395554 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89"] Nov 03 01:55:45.403003 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.402986 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.454534 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.454468 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89"] Nov 03 01:55:45.486816 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.486789 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nch9\" (UniqueName: \"kubernetes.io/projected/6557b888-0001-474f-9b39-508691839c30-kube-api-access-5nch9\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.486958 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.486825 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6557b888-0001-474f-9b39-508691839c30-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.486958 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.486847 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6557b888-0001-474f-9b39-508691839c30-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.486958 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.486913 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6557b888-0001-474f-9b39-508691839c30-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.486958 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.486942 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6557b888-0001-474f-9b39-508691839c30-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.487095 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.487033 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6557b888-0001-474f-9b39-508691839c30-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.487095 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.487078 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6557b888-0001-474f-9b39-508691839c30-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.487159 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.487119 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6557b888-0001-474f-9b39-508691839c30-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.487223 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.487165 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6557b888-0001-474f-9b39-508691839c30-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.502596 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.502566 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-fjxcb\"" Nov 03 01:55:45.504345 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.504311 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:45.588562 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.588530 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nch9\" (UniqueName: \"kubernetes.io/projected/6557b888-0001-474f-9b39-508691839c30-kube-api-access-5nch9\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.588737 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.588569 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6557b888-0001-474f-9b39-508691839c30-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.588737 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.588591 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6557b888-0001-474f-9b39-508691839c30-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.588737 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.588706 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6557b888-0001-474f-9b39-508691839c30-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.588908 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.588752 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6557b888-0001-474f-9b39-508691839c30-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.588908 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.588825 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6557b888-0001-474f-9b39-508691839c30-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.588908 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.588864 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6557b888-0001-474f-9b39-508691839c30-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.589065 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.588926 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6557b888-0001-474f-9b39-508691839c30-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.589065 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.588961 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6557b888-0001-474f-9b39-508691839c30-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.589065 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.588991 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6557b888-0001-474f-9b39-508691839c30-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.589327 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.589307 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6557b888-0001-474f-9b39-508691839c30-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.589426 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.589366 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6557b888-0001-474f-9b39-508691839c30-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.589489 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.589424 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6557b888-0001-474f-9b39-508691839c30-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.589787 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.589763 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6557b888-0001-474f-9b39-508691839c30-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.591095 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.591075 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6557b888-0001-474f-9b39-508691839c30-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.591461 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.591444 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6557b888-0001-474f-9b39-508691839c30-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.648509 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.648468 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6557b888-0001-474f-9b39-508691839c30-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.648650 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.648562 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nch9\" (UniqueName: \"kubernetes.io/projected/6557b888-0001-474f-9b39-508691839c30-kube-api-access-5nch9\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fzwb89\" (UID: \"6557b888-0001-474f-9b39-508691839c30\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.659495 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.659472 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:45.715842 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.715777 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:45.761360 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.761332 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-n9xw9"] Nov 03 01:55:45.766509 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.766484 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-n9xw9" Nov 03 01:55:45.820930 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.820899 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-n9xw9"] Nov 03 01:55:45.821545 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.821526 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qmnnn\"" Nov 03 01:55:45.891571 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.891538 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grxps\" (UniqueName: \"kubernetes.io/projected/ae1f04bb-4908-44a9-8701-93a7c7ae8f9f-kube-api-access-grxps\") pod \"authorino-57f7f8659f-n9xw9\" (UID: \"ae1f04bb-4908-44a9-8701-93a7c7ae8f9f\") " pod="kuadrant-system/authorino-57f7f8659f-n9xw9" Nov 03 01:55:45.902484 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.902454 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:45.992594 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:45.992511 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grxps\" (UniqueName: \"kubernetes.io/projected/ae1f04bb-4908-44a9-8701-93a7c7ae8f9f-kube-api-access-grxps\") pod \"authorino-57f7f8659f-n9xw9\" (UID: \"ae1f04bb-4908-44a9-8701-93a7c7ae8f9f\") " pod="kuadrant-system/authorino-57f7f8659f-n9xw9" Nov 03 01:55:46.049851 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:46.049814 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grxps\" (UniqueName: \"kubernetes.io/projected/ae1f04bb-4908-44a9-8701-93a7c7ae8f9f-kube-api-access-grxps\") pod \"authorino-57f7f8659f-n9xw9\" (UID: \"ae1f04bb-4908-44a9-8701-93a7c7ae8f9f\") " pod="kuadrant-system/authorino-57f7f8659f-n9xw9" Nov 03 01:55:46.059302 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:46.059273 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:46.100481 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:46.100449 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-n9xw9" Nov 03 01:55:46.131869 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:55:46.131828 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6557b888_0001_474f_9b39_508691839c30.slice/crio-9eb87d3a6ed766711353fe24bb0b9116446a44e358634067c02d743fde982240 WatchSource:0}: Error finding container 9eb87d3a6ed766711353fe24bb0b9116446a44e358634067c02d743fde982240: Status 404 returned error can't find the container with id 9eb87d3a6ed766711353fe24bb0b9116446a44e358634067c02d743fde982240 Nov 03 01:55:46.133582 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:46.133547 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89"] Nov 03 01:55:46.134412 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:46.134379 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:55:46.134515 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:46.134448 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:55:46.134515 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:46.134479 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:55:46.468890 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:55:46.468863 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae1f04bb_4908_44a9_8701_93a7c7ae8f9f.slice/crio-4529227d9ac60a021e386e206e3884cd216b7bf69f96711408cfe27fae56c5c7 WatchSource:0}: Error finding container 4529227d9ac60a021e386e206e3884cd216b7bf69f96711408cfe27fae56c5c7: Status 404 returned error can't find the container with id 4529227d9ac60a021e386e206e3884cd216b7bf69f96711408cfe27fae56c5c7 Nov 03 01:55:46.469296 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:46.469203 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-n9xw9"] Nov 03 01:55:46.604873 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:46.604722 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-n9xw9" event={"ID":"ae1f04bb-4908-44a9-8701-93a7c7ae8f9f","Type":"ContainerStarted","Data":"4529227d9ac60a021e386e206e3884cd216b7bf69f96711408cfe27fae56c5c7"} Nov 03 01:55:46.658968 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:46.606344 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" event={"ID":"6557b888-0001-474f-9b39-508691839c30","Type":"ContainerStarted","Data":"4813f14d01bcbd38de487995fe0a06732eecb5e9abb1596f8795d3cdcba81499"} Nov 03 01:55:46.658968 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:46.606391 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" event={"ID":"6557b888-0001-474f-9b39-508691839c30","Type":"ContainerStarted","Data":"9eb87d3a6ed766711353fe24bb0b9116446a44e358634067c02d743fde982240"} Nov 03 01:55:46.716721 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:46.716652 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:46.724758 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:46.724695 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" podStartSLOduration=1.724678253 podStartE2EDuration="1.724678253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:46.723795128 +0000 UTC m=+228.407904284" watchObservedRunningTime="2026-11-03 01:55:46.724678253 +0000 UTC m=+228.408787419" Nov 03 01:55:47.721069 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:47.721040 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:48.296683 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:48.296648 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-rq9cm"] Nov 03 01:55:48.302243 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:48.302221 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-rq9cm" Nov 03 01:55:48.361172 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:48.361147 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-rq9cm"] Nov 03 01:55:48.412577 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:48.412551 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5d2b\" (UniqueName: \"kubernetes.io/projected/ee4809b6-ec81-4d93-97c7-c484b46f1ae7-kube-api-access-b5d2b\") pod \"authorino-56c6c9ddd5-rq9cm\" (UID: \"ee4809b6-ec81-4d93-97c7-c484b46f1ae7\") " pod="kuadrant-system/authorino-56c6c9ddd5-rq9cm" Nov 03 01:55:48.513801 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:48.513773 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5d2b\" (UniqueName: \"kubernetes.io/projected/ee4809b6-ec81-4d93-97c7-c484b46f1ae7-kube-api-access-b5d2b\") pod \"authorino-56c6c9ddd5-rq9cm\" (UID: \"ee4809b6-ec81-4d93-97c7-c484b46f1ae7\") " pod="kuadrant-system/authorino-56c6c9ddd5-rq9cm" Nov 03 01:55:48.571493 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:48.571425 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5d2b\" (UniqueName: \"kubernetes.io/projected/ee4809b6-ec81-4d93-97c7-c484b46f1ae7-kube-api-access-b5d2b\") pod \"authorino-56c6c9ddd5-rq9cm\" (UID: \"ee4809b6-ec81-4d93-97c7-c484b46f1ae7\") " pod="kuadrant-system/authorino-56c6c9ddd5-rq9cm" Nov 03 01:55:48.613295 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:48.613271 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-rq9cm" Nov 03 01:55:48.615254 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:48.615227 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:48.616228 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:48.616209 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fzwb89" Nov 03 01:55:48.975887 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:55:48.975850 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee4809b6_ec81_4d93_97c7_c484b46f1ae7.slice/crio-bf2fa169b760835df556c0b75c7d1b5bd44c415e6715db3e682af63aee5ccfb1 WatchSource:0}: Error finding container bf2fa169b760835df556c0b75c7d1b5bd44c415e6715db3e682af63aee5ccfb1: Status 404 returned error can't find the container with id bf2fa169b760835df556c0b75c7d1b5bd44c415e6715db3e682af63aee5ccfb1 Nov 03 01:55:48.978034 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:48.978005 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-rq9cm"] Nov 03 01:55:49.621053 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:49.621019 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-rq9cm" event={"ID":"ee4809b6-ec81-4d93-97c7-c484b46f1ae7","Type":"ContainerStarted","Data":"bf2fa169b760835df556c0b75c7d1b5bd44c415e6715db3e682af63aee5ccfb1"} Nov 03 01:55:49.767303 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:49.767271 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:49.918792 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:49.918760 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:50.576083 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:50.576050 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt" Nov 03 01:55:50.796162 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:50.796127 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb"] Nov 03 01:55:50.796425 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:50.796402 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb" podUID="7861e771-80dd-4382-919a-ef02cb7d0892" containerName="manager" containerID="cri-o://9a27a092a5ebcb9ca39a28308f0ed007ea365f317b5d5a620c1eef7ccb16d99d" gracePeriod=10 Nov 03 01:55:51.346765 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:51.346737 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb" Nov 03 01:55:51.437771 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:51.437737 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7861e771-80dd-4382-919a-ef02cb7d0892-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7861e771-80dd-4382-919a-ef02cb7d0892-extensions-socket-volume\") pod \"7861e771-80dd-4382-919a-ef02cb7d0892\" (UID: \"7861e771-80dd-4382-919a-ef02cb7d0892\") " Nov 03 01:55:51.437933 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:51.437810 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7861e771-80dd-4382-919a-ef02cb7d0892-kube-api-access-x2t9s\" (UniqueName: \"kubernetes.io/projected/7861e771-80dd-4382-919a-ef02cb7d0892-kube-api-access-x2t9s\") pod \"7861e771-80dd-4382-919a-ef02cb7d0892\" (UID: \"7861e771-80dd-4382-919a-ef02cb7d0892\") " Nov 03 01:55:51.438292 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:51.438271 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7861e771-80dd-4382-919a-ef02cb7d0892-extensions-socket-volume" pod "7861e771-80dd-4382-919a-ef02cb7d0892" (UID: "7861e771-80dd-4382-919a-ef02cb7d0892"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:55:51.440057 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:51.440020 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7861e771-80dd-4382-919a-ef02cb7d0892-kube-api-access-x2t9s" pod "7861e771-80dd-4382-919a-ef02cb7d0892" (UID: "7861e771-80dd-4382-919a-ef02cb7d0892"). InnerVolumeSpecName "kube-api-access-x2t9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:55:51.539459 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:51.539382 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2t9s\" (UniqueName: \"kubernetes.io/projected/7861e771-80dd-4382-919a-ef02cb7d0892-kube-api-access-x2t9s\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:51.539459 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:51.539418 5463 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7861e771-80dd-4382-919a-ef02cb7d0892-extensions-socket-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:51.632385 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:51.632289 5463 generic.go:356] "Generic (PLEG): container finished" podID="7861e771-80dd-4382-919a-ef02cb7d0892" containerID="9a27a092a5ebcb9ca39a28308f0ed007ea365f317b5d5a620c1eef7ccb16d99d" exitCode=0 Nov 03 01:55:51.632385 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:51.632338 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb" event={"ID":"7861e771-80dd-4382-919a-ef02cb7d0892","Type":"ContainerDied","Data":"9a27a092a5ebcb9ca39a28308f0ed007ea365f317b5d5a620c1eef7ccb16d99d"} Nov 03 01:55:51.632385 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:51.632360 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb" Nov 03 01:55:51.632919 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:51.632409 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb" event={"ID":"7861e771-80dd-4382-919a-ef02cb7d0892","Type":"ContainerDied","Data":"ac59c197040384f3d5e846fadbd20128c9a5a6dc7c020dd0d1ad991a32c6d551"} Nov 03 01:55:51.632919 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:51.632443 5463 scope.go:122] "RemoveContainer" containerID="9a27a092a5ebcb9ca39a28308f0ed007ea365f317b5d5a620c1eef7ccb16d99d" Nov 03 01:55:51.641544 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:51.641525 5463 scope.go:122] "RemoveContainer" containerID="9a27a092a5ebcb9ca39a28308f0ed007ea365f317b5d5a620c1eef7ccb16d99d" Nov 03 01:55:51.641794 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:55:51.641775 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a27a092a5ebcb9ca39a28308f0ed007ea365f317b5d5a620c1eef7ccb16d99d\": container with ID starting with 9a27a092a5ebcb9ca39a28308f0ed007ea365f317b5d5a620c1eef7ccb16d99d not found: ID does not exist" containerID="9a27a092a5ebcb9ca39a28308f0ed007ea365f317b5d5a620c1eef7ccb16d99d" Nov 03 01:55:51.641838 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:51.641802 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a27a092a5ebcb9ca39a28308f0ed007ea365f317b5d5a620c1eef7ccb16d99d"} err="failed to get container status \"9a27a092a5ebcb9ca39a28308f0ed007ea365f317b5d5a620c1eef7ccb16d99d\": rpc error: code = NotFound desc = could not find container \"9a27a092a5ebcb9ca39a28308f0ed007ea365f317b5d5a620c1eef7ccb16d99d\": container with ID starting with 9a27a092a5ebcb9ca39a28308f0ed007ea365f317b5d5a620c1eef7ccb16d99d not found: ID does not exist" Nov 03 01:55:51.800502 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:51.800423 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb"] Nov 03 01:55:51.803896 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:51.803872 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-dhsqb"] Nov 03 01:55:51.989055 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:51.989017 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:52.443177 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:52.443144 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7861e771-80dd-4382-919a-ef02cb7d0892" path="/var/lib/kubelet/pods/7861e771-80dd-4382-919a-ef02cb7d0892/volumes" Nov 03 01:55:53.656817 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:53.656779 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-n9xw9" event={"ID":"ae1f04bb-4908-44a9-8701-93a7c7ae8f9f","Type":"ContainerStarted","Data":"e0662b5fd4107e337a3dc306032fffb162d952dcfbfa46a4a020a6dc3e7accaa"} Nov 03 01:55:53.658186 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:53.658163 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-rq9cm" event={"ID":"ee4809b6-ec81-4d93-97c7-c484b46f1ae7","Type":"ContainerStarted","Data":"19a78e8519d4de02f3c059a766c06678be594b613fb52952bb8881b2ca1f7ae3"} Nov 03 01:55:53.769896 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:53.769849 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-n9xw9" podStartSLOduration=2.490568831 podStartE2EDuration="8.769836517s" totalImagesPullingTime="6.279267686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:53.7690191 +0000 UTC m=+235.453128255" watchObservedRunningTime="2026-11-03 01:55:53.769836517 +0000 UTC m=+235.453945672" Nov 03 01:55:53.933557 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:53.933465 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-rq9cm" podStartSLOduration=2.156737775 podStartE2EDuration="5.93344673s" totalImagesPullingTime="3.776708955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:53.932951511 +0000 UTC m=+235.617060668" watchObservedRunningTime="2026-11-03 01:55:53.93344673 +0000 UTC m=+235.617555886" Nov 03 01:55:53.959609 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:53.959574 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-n9xw9"] Nov 03 01:55:55.668252 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:55.668213 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-n9xw9" podUID="ae1f04bb-4908-44a9-8701-93a7c7ae8f9f" containerName="authorino" containerID="cri-o://e0662b5fd4107e337a3dc306032fffb162d952dcfbfa46a4a020a6dc3e7accaa" gracePeriod=30 Nov 03 01:55:55.914110 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:55.914078 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-n9xw9" Nov 03 01:55:56.078003 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.077284 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1f04bb-4908-44a9-8701-93a7c7ae8f9f-kube-api-access-grxps\" (UniqueName: \"kubernetes.io/projected/ae1f04bb-4908-44a9-8701-93a7c7ae8f9f-kube-api-access-grxps\") pod \"ae1f04bb-4908-44a9-8701-93a7c7ae8f9f\" (UID: \"ae1f04bb-4908-44a9-8701-93a7c7ae8f9f\") " Nov 03 01:55:56.085205 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.085165 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1f04bb-4908-44a9-8701-93a7c7ae8f9f-kube-api-access-grxps" pod "ae1f04bb-4908-44a9-8701-93a7c7ae8f9f" (UID: "ae1f04bb-4908-44a9-8701-93a7c7ae8f9f"). InnerVolumeSpecName "kube-api-access-grxps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:55:56.178365 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.178339 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grxps\" (UniqueName: \"kubernetes.io/projected/ae1f04bb-4908-44a9-8701-93a7c7ae8f9f-kube-api-access-grxps\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:56.427691 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.427659 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn"] Nov 03 01:55:56.428043 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.428031 5463 cpu_manager.go:414] "containerMap: removing container" podUID="ae1f04bb-4908-44a9-8701-93a7c7ae8f9f" containerName="authorino" Nov 03 01:55:56.428088 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.428045 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1f04bb-4908-44a9-8701-93a7c7ae8f9f" containerName="authorino" Nov 03 01:55:56.428088 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.428063 5463 cpu_manager.go:414] "containerMap: removing container" podUID="7861e771-80dd-4382-919a-ef02cb7d0892" containerName="manager" Nov 03 01:55:56.428088 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.428068 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7861e771-80dd-4382-919a-ef02cb7d0892" containerName="manager" Nov 03 01:55:56.428179 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.428118 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1f04bb-4908-44a9-8701-93a7c7ae8f9f" containerName="authorino" Nov 03 01:55:56.428179 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.428126 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="7861e771-80dd-4382-919a-ef02cb7d0892" containerName="manager" Nov 03 01:55:56.431171 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.431153 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn" Nov 03 01:55:56.483925 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.483900 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 01:55:56.534181 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.534147 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 01:55:56.582404 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.582373 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f2d8c2b-cc81-4620-b32c-a0a056576e35-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-kb5pn\" (UID: \"2f2d8c2b-cc81-4620-b32c-a0a056576e35\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn" Nov 03 01:55:56.582404 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.582406 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f2d8c2b-cc81-4620-b32c-a0a056576e35-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-kb5pn\" (UID: \"2f2d8c2b-cc81-4620-b32c-a0a056576e35\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn" Nov 03 01:55:56.582578 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.582540 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f2d8c2b-cc81-4620-b32c-a0a056576e35-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-kb5pn\" (UID: \"2f2d8c2b-cc81-4620-b32c-a0a056576e35\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn" Nov 03 01:55:56.582578 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.582573 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j58xr\" (UniqueName: \"kubernetes.io/projected/2f2d8c2b-cc81-4620-b32c-a0a056576e35-kube-api-access-j58xr\") pod \"kserve-module-controller-manager-7566cf5fc8-kb5pn\" (UID: \"2f2d8c2b-cc81-4620-b32c-a0a056576e35\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn" Nov 03 01:55:56.636242 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.636213 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-pcdnw\"" Nov 03 01:55:56.636406 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.636363 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 01:55:56.637778 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.637760 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn"] Nov 03 01:55:56.674352 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.674326 5463 generic.go:356] "Generic (PLEG): container finished" podID="ae1f04bb-4908-44a9-8701-93a7c7ae8f9f" containerID="e0662b5fd4107e337a3dc306032fffb162d952dcfbfa46a4a020a6dc3e7accaa" exitCode=0 Nov 03 01:55:56.674695 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.674373 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-n9xw9" Nov 03 01:55:56.674695 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.674412 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-n9xw9" event={"ID":"ae1f04bb-4908-44a9-8701-93a7c7ae8f9f","Type":"ContainerDied","Data":"e0662b5fd4107e337a3dc306032fffb162d952dcfbfa46a4a020a6dc3e7accaa"} Nov 03 01:55:56.674695 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.674448 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-n9xw9" event={"ID":"ae1f04bb-4908-44a9-8701-93a7c7ae8f9f","Type":"ContainerDied","Data":"4529227d9ac60a021e386e206e3884cd216b7bf69f96711408cfe27fae56c5c7"} Nov 03 01:55:56.674695 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.674465 5463 scope.go:122] "RemoveContainer" containerID="e0662b5fd4107e337a3dc306032fffb162d952dcfbfa46a4a020a6dc3e7accaa" Nov 03 01:55:56.682649 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.682626 5463 scope.go:122] "RemoveContainer" containerID="e0662b5fd4107e337a3dc306032fffb162d952dcfbfa46a4a020a6dc3e7accaa" Nov 03 01:55:56.682938 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:55:56.682918 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0662b5fd4107e337a3dc306032fffb162d952dcfbfa46a4a020a6dc3e7accaa\": container with ID starting with e0662b5fd4107e337a3dc306032fffb162d952dcfbfa46a4a020a6dc3e7accaa not found: ID does not exist" containerID="e0662b5fd4107e337a3dc306032fffb162d952dcfbfa46a4a020a6dc3e7accaa" Nov 03 01:55:56.683022 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.682932 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f2d8c2b-cc81-4620-b32c-a0a056576e35-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-kb5pn\" (UID: \"2f2d8c2b-cc81-4620-b32c-a0a056576e35\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn" Nov 03 01:55:56.683022 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.682946 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0662b5fd4107e337a3dc306032fffb162d952dcfbfa46a4a020a6dc3e7accaa"} err="failed to get container status \"e0662b5fd4107e337a3dc306032fffb162d952dcfbfa46a4a020a6dc3e7accaa\": rpc error: code = NotFound desc = could not find container \"e0662b5fd4107e337a3dc306032fffb162d952dcfbfa46a4a020a6dc3e7accaa\": container with ID starting with e0662b5fd4107e337a3dc306032fffb162d952dcfbfa46a4a020a6dc3e7accaa not found: ID does not exist" Nov 03 01:55:56.683022 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.682961 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f2d8c2b-cc81-4620-b32c-a0a056576e35-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-kb5pn\" (UID: \"2f2d8c2b-cc81-4620-b32c-a0a056576e35\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn" Nov 03 01:55:56.683022 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.683016 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f2d8c2b-cc81-4620-b32c-a0a056576e35-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-kb5pn\" (UID: \"2f2d8c2b-cc81-4620-b32c-a0a056576e35\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn" Nov 03 01:55:56.683266 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.683042 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j58xr\" (UniqueName: \"kubernetes.io/projected/2f2d8c2b-cc81-4620-b32c-a0a056576e35-kube-api-access-j58xr\") pod \"kserve-module-controller-manager-7566cf5fc8-kb5pn\" (UID: \"2f2d8c2b-cc81-4620-b32c-a0a056576e35\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn" Nov 03 01:55:56.683365 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.683341 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f2d8c2b-cc81-4620-b32c-a0a056576e35-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-kb5pn\" (UID: \"2f2d8c2b-cc81-4620-b32c-a0a056576e35\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn" Nov 03 01:55:56.683431 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.683368 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f2d8c2b-cc81-4620-b32c-a0a056576e35-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-kb5pn\" (UID: \"2f2d8c2b-cc81-4620-b32c-a0a056576e35\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn" Nov 03 01:55:56.683911 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.683888 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f2d8c2b-cc81-4620-b32c-a0a056576e35-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-kb5pn\" (UID: \"2f2d8c2b-cc81-4620-b32c-a0a056576e35\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn" Nov 03 01:55:56.739472 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.739442 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j58xr\" (UniqueName: \"kubernetes.io/projected/2f2d8c2b-cc81-4620-b32c-a0a056576e35-kube-api-access-j58xr\") pod \"kserve-module-controller-manager-7566cf5fc8-kb5pn\" (UID: \"2f2d8c2b-cc81-4620-b32c-a0a056576e35\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn" Nov 03 01:55:56.744258 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.744237 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn" Nov 03 01:55:56.778378 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.778340 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:56.836969 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.836941 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-n9xw9"] Nov 03 01:55:56.842443 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.842422 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-n9xw9"] Nov 03 01:55:56.923674 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:56.923646 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:57.154601 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:55:57.154568 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2d8c2b_cc81_4620_b32c_a0a056576e35.slice/crio-f8bc5c9fc5807464cad31cf8060ec604ca09cf63efb7a5d706e9d08e512cb649 WatchSource:0}: Error finding container f8bc5c9fc5807464cad31cf8060ec604ca09cf63efb7a5d706e9d08e512cb649: Status 404 returned error can't find the container with id f8bc5c9fc5807464cad31cf8060ec604ca09cf63efb7a5d706e9d08e512cb649 Nov 03 01:55:57.156305 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:57.156285 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn"] Nov 03 01:55:57.681755 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:57.681723 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn" event={"ID":"2f2d8c2b-cc81-4620-b32c-a0a056576e35","Type":"ContainerStarted","Data":"f8bc5c9fc5807464cad31cf8060ec604ca09cf63efb7a5d706e9d08e512cb649"} Nov 03 01:55:58.441288 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:58.441256 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae1f04bb-4908-44a9-8701-93a7c7ae8f9f" path="/var/lib/kubelet/pods/ae1f04bb-4908-44a9-8701-93a7c7ae8f9f/volumes" Nov 03 01:55:59.214737 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:59.214701 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:59.360383 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:59.360330 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:59.890643 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:55:59.890608 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:00.035747 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:00.035708 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:01.704026 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:01.703991 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn" event={"ID":"2f2d8c2b-cc81-4620-b32c-a0a056576e35","Type":"ContainerStarted","Data":"7bcef677917e690beadfd233edac29adf1791f8c08b3e1f5d265cb98e74766cb"} Nov 03 01:56:01.704445 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:01.704138 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn" Nov 03 01:56:01.873729 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:01.873676 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn" podStartSLOduration=1.6851613300000001 podStartE2EDuration="5.87366003s" totalImagesPullingTime="4.1884987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:56:01.872751151 +0000 UTC m=+243.556860310" watchObservedRunningTime="2026-11-03 01:56:01.87366003 +0000 UTC m=+243.557769187" Nov 03 01:56:02.880063 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:02.880029 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:03.025031 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:03.024997 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:04.778744 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:04.778716 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:04.933973 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:04.933927 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:05.379808 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:05.379777 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:05.526880 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:05.526851 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:09.568735 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:09.568698 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:09.716214 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:09.716162 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:09.985529 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:09.985498 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:10.205434 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:10.205404 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:10.381159 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:10.381078 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:10.530851 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:10.530821 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:12.773614 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:12.773576 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:12.943501 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:12.943467 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:13.005072 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:56:13.005041 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 01:56:13.094676 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:13.094603 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:14.150233 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.150187 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp"] Nov 03 01:56:14.153699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.153683 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp" Nov 03 01:56:14.240124 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.240092 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2d46175-1dc4-4838-a3c0-19f879a91404-metrics-certs\") pod \"jobset-controller-manager-6cf78bbdc8-vl8wp\" (UID: \"b2d46175-1dc4-4838-a3c0-19f879a91404\") " pod="openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp" Nov 03 01:56:14.240305 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.240128 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2467m\" (UniqueName: \"kubernetes.io/projected/b2d46175-1dc4-4838-a3c0-19f879a91404-kube-api-access-2467m\") pod \"jobset-controller-manager-6cf78bbdc8-vl8wp\" (UID: \"b2d46175-1dc4-4838-a3c0-19f879a91404\") " pod="openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp" Nov 03 01:56:14.240305 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.240236 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b2d46175-1dc4-4838-a3c0-19f879a91404-manager-config\") pod \"jobset-controller-manager-6cf78bbdc8-vl8wp\" (UID: \"b2d46175-1dc4-4838-a3c0-19f879a91404\") " pod="openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp" Nov 03 01:56:14.240305 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.240287 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2d46175-1dc4-4838-a3c0-19f879a91404-cert\") pod \"jobset-controller-manager-6cf78bbdc8-vl8wp\" (UID: \"b2d46175-1dc4-4838-a3c0-19f879a91404\") " pod="openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp" Nov 03 01:56:14.259326 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.259287 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 01:56:14.259326 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.259288 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 01:56:14.259502 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.259325 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 01:56:14.259502 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.259384 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-lvbjl\"" Nov 03 01:56:14.259502 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.259331 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 01:56:14.259599 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.259517 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:56:14.260745 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.260726 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp"] Nov 03 01:56:14.340749 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.340720 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b2d46175-1dc4-4838-a3c0-19f879a91404-manager-config\") pod \"jobset-controller-manager-6cf78bbdc8-vl8wp\" (UID: \"b2d46175-1dc4-4838-a3c0-19f879a91404\") " pod="openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp" Nov 03 01:56:14.340899 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.340774 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2d46175-1dc4-4838-a3c0-19f879a91404-cert\") pod \"jobset-controller-manager-6cf78bbdc8-vl8wp\" (UID: \"b2d46175-1dc4-4838-a3c0-19f879a91404\") " pod="openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp" Nov 03 01:56:14.340899 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.340846 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2d46175-1dc4-4838-a3c0-19f879a91404-metrics-certs\") pod \"jobset-controller-manager-6cf78bbdc8-vl8wp\" (UID: \"b2d46175-1dc4-4838-a3c0-19f879a91404\") " pod="openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp" Nov 03 01:56:14.340899 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.340879 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2467m\" (UniqueName: \"kubernetes.io/projected/b2d46175-1dc4-4838-a3c0-19f879a91404-kube-api-access-2467m\") pod \"jobset-controller-manager-6cf78bbdc8-vl8wp\" (UID: \"b2d46175-1dc4-4838-a3c0-19f879a91404\") " pod="openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp" Nov 03 01:56:14.341449 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.341428 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b2d46175-1dc4-4838-a3c0-19f879a91404-manager-config\") pod \"jobset-controller-manager-6cf78bbdc8-vl8wp\" (UID: \"b2d46175-1dc4-4838-a3c0-19f879a91404\") " pod="openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp" Nov 03 01:56:14.343490 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.343461 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2d46175-1dc4-4838-a3c0-19f879a91404-metrics-certs\") pod \"jobset-controller-manager-6cf78bbdc8-vl8wp\" (UID: \"b2d46175-1dc4-4838-a3c0-19f879a91404\") " pod="openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp" Nov 03 01:56:14.343590 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.343465 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2d46175-1dc4-4838-a3c0-19f879a91404-cert\") pod \"jobset-controller-manager-6cf78bbdc8-vl8wp\" (UID: \"b2d46175-1dc4-4838-a3c0-19f879a91404\") " pod="openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp" Nov 03 01:56:14.397678 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.397650 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2467m\" (UniqueName: \"kubernetes.io/projected/b2d46175-1dc4-4838-a3c0-19f879a91404-kube-api-access-2467m\") pod \"jobset-controller-manager-6cf78bbdc8-vl8wp\" (UID: \"b2d46175-1dc4-4838-a3c0-19f879a91404\") " pod="openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp" Nov 03 01:56:14.466914 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.466847 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp" Nov 03 01:56:14.832816 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:56:14.832781 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d46175_1dc4_4838_a3c0_19f879a91404.slice/crio-dec0c2659a7dd00bcf3a6c9acf18c0f72d49fdc5b948e6bd3fe2ebad75899b04 WatchSource:0}: Error finding container dec0c2659a7dd00bcf3a6c9acf18c0f72d49fdc5b948e6bd3fe2ebad75899b04: Status 404 returned error can't find the container with id dec0c2659a7dd00bcf3a6c9acf18c0f72d49fdc5b948e6bd3fe2ebad75899b04 Nov 03 01:56:14.834349 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:14.834328 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp"] Nov 03 01:56:15.774887 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:15.774846 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp" event={"ID":"b2d46175-1dc4-4838-a3c0-19f879a91404","Type":"ContainerStarted","Data":"dec0c2659a7dd00bcf3a6c9acf18c0f72d49fdc5b948e6bd3fe2ebad75899b04"} Nov 03 01:56:16.383316 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:16.383284 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:16.519540 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:16.519503 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:18.814781 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:18.814743 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:18.964055 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:18.964019 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:20.801467 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:20.801434 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp" event={"ID":"b2d46175-1dc4-4838-a3c0-19f879a91404","Type":"ContainerStarted","Data":"a12d1d3bc8b16b40c882b762f26cd24823bfdad910cdf414eb16c48a0911fb03"} Nov 03 01:56:20.801818 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:20.801610 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp" Nov 03 01:56:20.915467 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:20.915413 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp" podStartSLOduration=1.5919676470000002 podStartE2EDuration="6.915399807s" totalImagesPullingTime="5.32343216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:56:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:56:20.915091333 +0000 UTC m=+262.599200490" watchObservedRunningTime="2026-11-03 01:56:20.915399807 +0000 UTC m=+262.599508963" Nov 03 01:56:22.710572 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:22.710539 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kb5pn" Nov 03 01:56:25.593776 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:25.593742 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:25.743557 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:25.743521 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:26.398965 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:26.398931 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:31.811006 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:31.810973 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6cf78bbdc8-vl8wp" Nov 03 01:56:43.006453 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:56:43.006374 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 01:56:44.975126 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:44.975085 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:49.459377 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:49.459342 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:49.620234 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:49.620183 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:49.879575 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:49.879487 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:50.044779 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:50.044732 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:58.461720 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:58.461690 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 01:56:58.462428 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:58.462411 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 01:56:58.908675 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:58.908624 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:59.134959 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:56:59.134917 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:02.378845 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:02.378806 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:02.536634 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:02.536589 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:02.691227 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:02.691185 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:02.967506 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:02.967387 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:03.129066 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:03.129030 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:03.283807 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:03.283737 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:03.445275 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:03.445240 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:04.591697 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:04.591664 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:05.153423 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:05.153395 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:09.500313 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:09.500276 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:09.649761 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:09.649729 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:09.784807 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:09.784737 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:09.926350 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:09.926321 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:10.069816 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:10.069748 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:10.222256 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:10.222222 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:10.370126 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:10.370034 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:10.513169 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:10.513140 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:10.654326 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:10.654298 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:13.005273 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:57:13.005222 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 01:57:24.551700 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:24.551663 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:26.992169 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:26.992131 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:27.148668 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:27.148634 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:29.066094 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:29.066058 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:29.221989 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:29.221951 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:29.364126 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:29.364051 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:29.512037 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:29.512005 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:43.006701 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:57:43.006659 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 01:57:44.211235 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:44.211205 5463 generic.go:356] "Generic (PLEG): container finished" podID="83d426e2-ed63-493c-871e-aa5d62476d77" containerID="08d9a0691aa88b6a9ce41589b8d8fb008c907a3d2ae04c1de7d2c8d3a95871df" exitCode=0 Nov 03 01:57:44.211653 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:44.211264 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" event={"ID":"83d426e2-ed63-493c-871e-aa5d62476d77","Type":"ContainerDied","Data":"08d9a0691aa88b6a9ce41589b8d8fb008c907a3d2ae04c1de7d2c8d3a95871df"} Nov 03 01:57:44.211653 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:44.211614 5463 scope.go:122] "RemoveContainer" containerID="08d9a0691aa88b6a9ce41589b8d8fb008c907a3d2ae04c1de7d2c8d3a95871df" Nov 03 01:57:44.212674 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:44.212644 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:57:44.212747 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:44.212725 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:57:44.212781 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:44.212760 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:57:44.646737 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:44.646710 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:44.965610 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:44.965480 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kc5zs"] Nov 03 01:57:44.973709 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:44.973675 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:44.974265 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:44.974246 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-kc5zs" Nov 03 01:57:45.012612 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.012586 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kc5zs"] Nov 03 01:57:45.074145 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.074100 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcwf9\" (UniqueName: \"kubernetes.io/projected/a7049888-ff04-45ae-b999-447d02db686b-kube-api-access-gcwf9\") pod \"authorino-6f6944bcdd-kc5zs\" (UID: \"a7049888-ff04-45ae-b999-447d02db686b\") " pod="kuadrant-system/authorino-6f6944bcdd-kc5zs" Nov 03 01:57:45.174737 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.174694 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcwf9\" (UniqueName: \"kubernetes.io/projected/a7049888-ff04-45ae-b999-447d02db686b-kube-api-access-gcwf9\") pod \"authorino-6f6944bcdd-kc5zs\" (UID: \"a7049888-ff04-45ae-b999-447d02db686b\") " pod="kuadrant-system/authorino-6f6944bcdd-kc5zs" Nov 03 01:57:45.220309 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.220224 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" event={"ID":"83d426e2-ed63-493c-871e-aa5d62476d77","Type":"ContainerStarted","Data":"29c0b4e7f860188489ef00c74f9ee87ef484aa0145812bf2d518ffc68627a1df"} Nov 03 01:57:45.231887 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.231857 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcwf9\" (UniqueName: \"kubernetes.io/projected/a7049888-ff04-45ae-b999-447d02db686b-kube-api-access-gcwf9\") pod \"authorino-6f6944bcdd-kc5zs\" (UID: \"a7049888-ff04-45ae-b999-447d02db686b\") " pod="kuadrant-system/authorino-6f6944bcdd-kc5zs" Nov 03 01:57:45.285344 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.285310 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-kc5zs" Nov 03 01:57:45.337838 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.337807 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kc5zs"] Nov 03 01:57:45.377635 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.377543 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6cfdb88454-fnjmn"] Nov 03 01:57:45.380220 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.379678 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m6hqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 03 01:57:45.380220 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.379731 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" podUID="83d426e2-ed63-493c-871e-aa5d62476d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 03 01:57:45.381350 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.380774 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:57:45.381350 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.380912 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cfdb88454-fnjmn" Nov 03 01:57:45.433629 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.433600 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cfdb88454-fnjmn"] Nov 03 01:57:45.578381 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.578296 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np8ds\" (UniqueName: \"kubernetes.io/projected/24cac5dc-ec95-4917-88ea-f4fbed528619-kube-api-access-np8ds\") pod \"authorino-6cfdb88454-fnjmn\" (UID: \"24cac5dc-ec95-4917-88ea-f4fbed528619\") " pod="kuadrant-system/authorino-6cfdb88454-fnjmn" Nov 03 01:57:45.661215 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:57:45.660303 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7049888_ff04_45ae_b999_447d02db686b.slice/crio-fe4b06db20abd646a3eea4d8cdc52d5540ad72d8f353f2ddb25722256b544e06 WatchSource:0}: Error finding container fe4b06db20abd646a3eea4d8cdc52d5540ad72d8f353f2ddb25722256b544e06: Status 404 returned error can't find the container with id fe4b06db20abd646a3eea4d8cdc52d5540ad72d8f353f2ddb25722256b544e06 Nov 03 01:57:45.661215 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.660532 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cfdb88454-fnjmn"] Nov 03 01:57:45.661215 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:57:45.660778 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-np8ds], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6cfdb88454-fnjmn" podUID="24cac5dc-ec95-4917-88ea-f4fbed528619" Nov 03 01:57:45.676224 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.661725 5463 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:57:45.676914 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.676882 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kc5zs"] Nov 03 01:57:45.677892 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.677871 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-78dd457b-trpxb"] Nov 03 01:57:45.679184 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.679159 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np8ds\" (UniqueName: \"kubernetes.io/projected/24cac5dc-ec95-4917-88ea-f4fbed528619-kube-api-access-np8ds\") pod \"authorino-6cfdb88454-fnjmn\" (UID: \"24cac5dc-ec95-4917-88ea-f4fbed528619\") " pod="kuadrant-system/authorino-6cfdb88454-fnjmn" Nov 03 01:57:45.680408 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.680394 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78dd457b-trpxb" Nov 03 01:57:45.732665 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.732631 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78dd457b-trpxb"] Nov 03 01:57:45.734090 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.734070 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 01:57:45.737964 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.737940 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np8ds\" (UniqueName: \"kubernetes.io/projected/24cac5dc-ec95-4917-88ea-f4fbed528619-kube-api-access-np8ds\") pod \"authorino-6cfdb88454-fnjmn\" (UID: \"24cac5dc-ec95-4917-88ea-f4fbed528619\") " pod="kuadrant-system/authorino-6cfdb88454-fnjmn" Nov 03 01:57:45.779722 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.779696 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdxj8\" (UniqueName: \"kubernetes.io/projected/651e5b21-a142-4a71-b744-d8f198e32183-kube-api-access-tdxj8\") pod \"authorino-78dd457b-trpxb\" (UID: \"651e5b21-a142-4a71-b744-d8f198e32183\") " pod="kuadrant-system/authorino-78dd457b-trpxb" Nov 03 01:57:45.779843 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.779751 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/651e5b21-a142-4a71-b744-d8f198e32183-tls-cert\") pod \"authorino-78dd457b-trpxb\" (UID: \"651e5b21-a142-4a71-b744-d8f198e32183\") " pod="kuadrant-system/authorino-78dd457b-trpxb" Nov 03 01:57:45.880896 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.880820 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/651e5b21-a142-4a71-b744-d8f198e32183-tls-cert\") pod \"authorino-78dd457b-trpxb\" (UID: \"651e5b21-a142-4a71-b744-d8f198e32183\") " pod="kuadrant-system/authorino-78dd457b-trpxb" Nov 03 01:57:45.880896 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.880881 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdxj8\" (UniqueName: \"kubernetes.io/projected/651e5b21-a142-4a71-b744-d8f198e32183-kube-api-access-tdxj8\") pod \"authorino-78dd457b-trpxb\" (UID: \"651e5b21-a142-4a71-b744-d8f198e32183\") " pod="kuadrant-system/authorino-78dd457b-trpxb" Nov 03 01:57:45.883434 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.883410 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/651e5b21-a142-4a71-b744-d8f198e32183-tls-cert\") pod \"authorino-78dd457b-trpxb\" (UID: \"651e5b21-a142-4a71-b744-d8f198e32183\") " pod="kuadrant-system/authorino-78dd457b-trpxb" Nov 03 01:57:45.938086 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.938061 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdxj8\" (UniqueName: \"kubernetes.io/projected/651e5b21-a142-4a71-b744-d8f198e32183-kube-api-access-tdxj8\") pod \"authorino-78dd457b-trpxb\" (UID: \"651e5b21-a142-4a71-b744-d8f198e32183\") " pod="kuadrant-system/authorino-78dd457b-trpxb" Nov 03 01:57:45.992936 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:45.992911 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78dd457b-trpxb" Nov 03 01:57:46.226394 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:46.226354 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-kc5zs" event={"ID":"a7049888-ff04-45ae-b999-447d02db686b","Type":"ContainerStarted","Data":"fe4b06db20abd646a3eea4d8cdc52d5540ad72d8f353f2ddb25722256b544e06"} Nov 03 01:57:46.226394 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:46.226388 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cfdb88454-fnjmn" Nov 03 01:57:46.233125 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:46.233107 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cfdb88454-fnjmn" Nov 03 01:57:46.356954 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:46.356845 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78dd457b-trpxb"] Nov 03 01:57:46.376779 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:57:46.376750 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod651e5b21_a142_4a71_b744_d8f198e32183.slice/crio-ea5873c20af686291c33d3f35985c4d85876eafa76de688088286f3cbc3e2cb8 WatchSource:0}: Error finding container ea5873c20af686291c33d3f35985c4d85876eafa76de688088286f3cbc3e2cb8: Status 404 returned error can't find the container with id ea5873c20af686291c33d3f35985c4d85876eafa76de688088286f3cbc3e2cb8 Nov 03 01:57:46.380380 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:46.380357 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m6hqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 03 01:57:46.380473 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:46.380397 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" podUID="83d426e2-ed63-493c-871e-aa5d62476d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 03 01:57:46.386412 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:46.386391 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24cac5dc-ec95-4917-88ea-f4fbed528619-kube-api-access-np8ds\" (UniqueName: \"kubernetes.io/projected/24cac5dc-ec95-4917-88ea-f4fbed528619-kube-api-access-np8ds\") pod \"24cac5dc-ec95-4917-88ea-f4fbed528619\" (UID: \"24cac5dc-ec95-4917-88ea-f4fbed528619\") " Nov 03 01:57:46.388286 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:46.388266 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24cac5dc-ec95-4917-88ea-f4fbed528619-kube-api-access-np8ds" pod "24cac5dc-ec95-4917-88ea-f4fbed528619" (UID: "24cac5dc-ec95-4917-88ea-f4fbed528619"). InnerVolumeSpecName "kube-api-access-np8ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:57:46.487721 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:46.487662 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np8ds\" (UniqueName: \"kubernetes.io/projected/24cac5dc-ec95-4917-88ea-f4fbed528619-kube-api-access-np8ds\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:47.233020 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:47.232993 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cfdb88454-fnjmn" Nov 03 01:57:47.233526 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:47.233018 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78dd457b-trpxb" event={"ID":"651e5b21-a142-4a71-b744-d8f198e32183","Type":"ContainerStarted","Data":"ea5873c20af686291c33d3f35985c4d85876eafa76de688088286f3cbc3e2cb8"} Nov 03 01:57:47.379855 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:47.379821 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m6hqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 03 01:57:47.380045 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:47.379873 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" podUID="83d426e2-ed63-493c-871e-aa5d62476d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 03 01:57:47.477987 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:47.477956 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cfdb88454-fnjmn"] Nov 03 01:57:47.479601 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:47.479572 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6cfdb88454-fnjmn"] Nov 03 01:57:48.240696 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:48.240657 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78dd457b-trpxb" event={"ID":"651e5b21-a142-4a71-b744-d8f198e32183","Type":"ContainerStarted","Data":"a123dc2bb649480a6d7f68e101ced74bcc2c56efffcc732dc3ba0c4daf4716eb"} Nov 03 01:57:48.242853 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:48.242825 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-kc5zs" event={"ID":"a7049888-ff04-45ae-b999-447d02db686b","Type":"ContainerStarted","Data":"95cddf9b311d70bbab918cf1a201f911001224c9448e76e6c4c969794e38f072"} Nov 03 01:57:48.243001 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:48.242932 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-kc5zs" podUID="a7049888-ff04-45ae-b999-447d02db686b" containerName="authorino" containerID="cri-o://95cddf9b311d70bbab918cf1a201f911001224c9448e76e6c4c969794e38f072" gracePeriod=30 Nov 03 01:57:48.376404 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:48.376341 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-78dd457b-trpxb" podStartSLOduration=2.070485036 podStartE2EDuration="3.376321643s" totalImagesPullingTime="1.305836607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:57:48.373594101 +0000 UTC m=+350.057703260" watchObservedRunningTime="2026-11-03 01:57:48.376321643 +0000 UTC m=+350.060430799" Nov 03 01:57:48.393818 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:48.393787 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m6hqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 03 01:57:48.394239 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:48.394176 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" podUID="83d426e2-ed63-493c-871e-aa5d62476d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 03 01:57:48.395487 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:48.394662 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-rq9cm"] Nov 03 01:57:48.395487 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:48.394920 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-rq9cm" podUID="ee4809b6-ec81-4d93-97c7-c484b46f1ae7" containerName="authorino" containerID="cri-o://19a78e8519d4de02f3c059a766c06678be594b613fb52952bb8881b2ca1f7ae3" gracePeriod=30 Nov 03 01:57:48.444824 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:48.444792 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24cac5dc-ec95-4917-88ea-f4fbed528619" path="/var/lib/kubelet/pods/24cac5dc-ec95-4917-88ea-f4fbed528619/volumes" Nov 03 01:57:48.500395 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:48.499954 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-kc5zs" podStartSLOduration=2.682884567 podStartE2EDuration="4.499935476s" totalImagesPullingTime="1.817050909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:57:48.498396082 +0000 UTC m=+350.182505250" watchObservedRunningTime="2026-11-03 01:57:48.499935476 +0000 UTC m=+350.184044631" Nov 03 01:57:48.603537 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:48.603237 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-kc5zs" Nov 03 01:57:48.714325 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:48.714299 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-rq9cm" Nov 03 01:57:48.809362 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:48.809281 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7049888-ff04-45ae-b999-447d02db686b-kube-api-access-gcwf9\" (UniqueName: \"kubernetes.io/projected/a7049888-ff04-45ae-b999-447d02db686b-kube-api-access-gcwf9\") pod \"a7049888-ff04-45ae-b999-447d02db686b\" (UID: \"a7049888-ff04-45ae-b999-447d02db686b\") " Nov 03 01:57:48.811574 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:48.811546 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7049888-ff04-45ae-b999-447d02db686b-kube-api-access-gcwf9" pod "a7049888-ff04-45ae-b999-447d02db686b" (UID: "a7049888-ff04-45ae-b999-447d02db686b"). InnerVolumeSpecName "kube-api-access-gcwf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:57:48.910569 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:48.910490 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee4809b6-ec81-4d93-97c7-c484b46f1ae7-kube-api-access-b5d2b\" (UniqueName: \"kubernetes.io/projected/ee4809b6-ec81-4d93-97c7-c484b46f1ae7-kube-api-access-b5d2b\") pod \"ee4809b6-ec81-4d93-97c7-c484b46f1ae7\" (UID: \"ee4809b6-ec81-4d93-97c7-c484b46f1ae7\") " Nov 03 01:57:48.911006 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:48.910909 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcwf9\" (UniqueName: \"kubernetes.io/projected/a7049888-ff04-45ae-b999-447d02db686b-kube-api-access-gcwf9\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:48.912831 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:48.912809 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4809b6-ec81-4d93-97c7-c484b46f1ae7-kube-api-access-b5d2b" pod "ee4809b6-ec81-4d93-97c7-c484b46f1ae7" (UID: "ee4809b6-ec81-4d93-97c7-c484b46f1ae7"). InnerVolumeSpecName "kube-api-access-b5d2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:57:49.011905 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.011872 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5d2b\" (UniqueName: \"kubernetes.io/projected/ee4809b6-ec81-4d93-97c7-c484b46f1ae7-kube-api-access-b5d2b\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:49.269215 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.268211 5463 generic.go:356] "Generic (PLEG): container finished" podID="ee4809b6-ec81-4d93-97c7-c484b46f1ae7" containerID="19a78e8519d4de02f3c059a766c06678be594b613fb52952bb8881b2ca1f7ae3" exitCode=0 Nov 03 01:57:49.269215 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.268399 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-rq9cm" event={"ID":"ee4809b6-ec81-4d93-97c7-c484b46f1ae7","Type":"ContainerDied","Data":"19a78e8519d4de02f3c059a766c06678be594b613fb52952bb8881b2ca1f7ae3"} Nov 03 01:57:49.269215 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.268444 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-rq9cm" event={"ID":"ee4809b6-ec81-4d93-97c7-c484b46f1ae7","Type":"ContainerDied","Data":"bf2fa169b760835df556c0b75c7d1b5bd44c415e6715db3e682af63aee5ccfb1"} Nov 03 01:57:49.269215 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.268469 5463 scope.go:122] "RemoveContainer" containerID="19a78e8519d4de02f3c059a766c06678be594b613fb52952bb8881b2ca1f7ae3" Nov 03 01:57:49.269215 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.268625 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-rq9cm" Nov 03 01:57:49.274785 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.274751 5463 generic.go:356] "Generic (PLEG): container finished" podID="a7049888-ff04-45ae-b999-447d02db686b" containerID="95cddf9b311d70bbab918cf1a201f911001224c9448e76e6c4c969794e38f072" exitCode=0 Nov 03 01:57:49.274905 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.274875 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-kc5zs" event={"ID":"a7049888-ff04-45ae-b999-447d02db686b","Type":"ContainerDied","Data":"95cddf9b311d70bbab918cf1a201f911001224c9448e76e6c4c969794e38f072"} Nov 03 01:57:49.274967 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.274930 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-kc5zs" event={"ID":"a7049888-ff04-45ae-b999-447d02db686b","Type":"ContainerDied","Data":"fe4b06db20abd646a3eea4d8cdc52d5540ad72d8f353f2ddb25722256b544e06"} Nov 03 01:57:49.275155 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.275119 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-kc5zs" Nov 03 01:57:49.282488 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.282471 5463 scope.go:122] "RemoveContainer" containerID="19a78e8519d4de02f3c059a766c06678be594b613fb52952bb8881b2ca1f7ae3" Nov 03 01:57:49.282760 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:57:49.282732 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19a78e8519d4de02f3c059a766c06678be594b613fb52952bb8881b2ca1f7ae3\": container with ID starting with 19a78e8519d4de02f3c059a766c06678be594b613fb52952bb8881b2ca1f7ae3 not found: ID does not exist" containerID="19a78e8519d4de02f3c059a766c06678be594b613fb52952bb8881b2ca1f7ae3" Nov 03 01:57:49.282811 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.282775 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a78e8519d4de02f3c059a766c06678be594b613fb52952bb8881b2ca1f7ae3"} err="failed to get container status \"19a78e8519d4de02f3c059a766c06678be594b613fb52952bb8881b2ca1f7ae3\": rpc error: code = NotFound desc = could not find container \"19a78e8519d4de02f3c059a766c06678be594b613fb52952bb8881b2ca1f7ae3\": container with ID starting with 19a78e8519d4de02f3c059a766c06678be594b613fb52952bb8881b2ca1f7ae3 not found: ID does not exist" Nov 03 01:57:49.282811 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.282799 5463 scope.go:122] "RemoveContainer" containerID="95cddf9b311d70bbab918cf1a201f911001224c9448e76e6c4c969794e38f072" Nov 03 01:57:49.290948 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.290933 5463 scope.go:122] "RemoveContainer" containerID="95cddf9b311d70bbab918cf1a201f911001224c9448e76e6c4c969794e38f072" Nov 03 01:57:49.291223 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:57:49.291182 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95cddf9b311d70bbab918cf1a201f911001224c9448e76e6c4c969794e38f072\": container with ID starting with 95cddf9b311d70bbab918cf1a201f911001224c9448e76e6c4c969794e38f072 not found: ID does not exist" containerID="95cddf9b311d70bbab918cf1a201f911001224c9448e76e6c4c969794e38f072" Nov 03 01:57:49.291268 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.291234 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95cddf9b311d70bbab918cf1a201f911001224c9448e76e6c4c969794e38f072"} err="failed to get container status \"95cddf9b311d70bbab918cf1a201f911001224c9448e76e6c4c969794e38f072\": rpc error: code = NotFound desc = could not find container \"95cddf9b311d70bbab918cf1a201f911001224c9448e76e6c4c969794e38f072\": container with ID starting with 95cddf9b311d70bbab918cf1a201f911001224c9448e76e6c4c969794e38f072 not found: ID does not exist" Nov 03 01:57:49.380345 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.380267 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m6hqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 03 01:57:49.380345 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.380328 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" podUID="83d426e2-ed63-493c-871e-aa5d62476d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 03 01:57:49.435274 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.435242 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kc5zs"] Nov 03 01:57:49.436820 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.436798 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kc5zs"] Nov 03 01:57:49.546790 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.546724 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-rq9cm"] Nov 03 01:57:49.555937 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:49.555879 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-rq9cm"] Nov 03 01:57:50.383502 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:50.383474 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:57:50.383895 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:50.383849 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:57:50.384528 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:50.384511 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 01:57:50.441367 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:50.441345 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7049888-ff04-45ae-b999-447d02db686b" path="/var/lib/kubelet/pods/a7049888-ff04-45ae-b999-447d02db686b/volumes" Nov 03 01:57:50.441660 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:50.441648 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee4809b6-ec81-4d93-97c7-c484b46f1ae7" path="/var/lib/kubelet/pods/ee4809b6-ec81-4d93-97c7-c484b46f1ae7/volumes" Nov 03 01:57:53.401522 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:53.401489 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:53.544128 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:57:53.544082 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:05.480589 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:05.480548 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:13.005978 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:58:13.005895 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 01:58:43.005679 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:58:43.005632 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 01:58:43.583060 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.583032 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-jqswh"] Nov 03 01:58:43.583423 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.583409 5463 cpu_manager.go:414] "containerMap: removing container" podUID="ee4809b6-ec81-4d93-97c7-c484b46f1ae7" containerName="authorino" Nov 03 01:58:43.583480 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.583425 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4809b6-ec81-4d93-97c7-c484b46f1ae7" containerName="authorino" Nov 03 01:58:43.583480 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.583438 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a7049888-ff04-45ae-b999-447d02db686b" containerName="authorino" Nov 03 01:58:43.583480 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.583443 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7049888-ff04-45ae-b999-447d02db686b" containerName="authorino" Nov 03 01:58:43.583588 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.583513 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee4809b6-ec81-4d93-97c7-c484b46f1ae7" containerName="authorino" Nov 03 01:58:43.583588 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.583521 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7049888-ff04-45ae-b999-447d02db686b" containerName="authorino" Nov 03 01:58:43.587115 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.587088 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-jqswh" Nov 03 01:58:43.588587 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.588560 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:43.639254 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.639225 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-jqswh"] Nov 03 01:58:43.671441 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.671410 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/650460ed-84e8-4f97-9f92-82f5854dd6c7-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-jqswh\" (UID: \"650460ed-84e8-4f97-9f92-82f5854dd6c7\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-jqswh" Nov 03 01:58:43.671441 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.671442 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78khm\" (UniqueName: \"kubernetes.io/projected/650460ed-84e8-4f97-9f92-82f5854dd6c7-kube-api-access-78khm\") pod \"ai-gateway-controller-5d8dddb676-jqswh\" (UID: \"650460ed-84e8-4f97-9f92-82f5854dd6c7\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-jqswh" Nov 03 01:58:43.688016 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.687989 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 01:58:43.690419 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.690400 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 01:58:43.690496 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.690414 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-r8xmb\"" Nov 03 01:58:43.772095 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.772063 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/650460ed-84e8-4f97-9f92-82f5854dd6c7-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-jqswh\" (UID: \"650460ed-84e8-4f97-9f92-82f5854dd6c7\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-jqswh" Nov 03 01:58:43.772095 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.772095 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78khm\" (UniqueName: \"kubernetes.io/projected/650460ed-84e8-4f97-9f92-82f5854dd6c7-kube-api-access-78khm\") pod \"ai-gateway-controller-5d8dddb676-jqswh\" (UID: \"650460ed-84e8-4f97-9f92-82f5854dd6c7\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-jqswh" Nov 03 01:58:43.774554 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.774532 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/650460ed-84e8-4f97-9f92-82f5854dd6c7-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-jqswh\" (UID: \"650460ed-84e8-4f97-9f92-82f5854dd6c7\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-jqswh" Nov 03 01:58:43.789013 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.788981 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:43.837084 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.837024 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78khm\" (UniqueName: \"kubernetes.io/projected/650460ed-84e8-4f97-9f92-82f5854dd6c7-kube-api-access-78khm\") pod \"ai-gateway-controller-5d8dddb676-jqswh\" (UID: \"650460ed-84e8-4f97-9f92-82f5854dd6c7\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-jqswh" Nov 03 01:58:43.900929 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:43.900900 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-jqswh" Nov 03 01:58:44.269825 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:44.269760 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-jqswh"] Nov 03 01:58:44.537943 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:44.537863 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-jqswh" event={"ID":"650460ed-84e8-4f97-9f92-82f5854dd6c7","Type":"ContainerStarted","Data":"69dfce93324250796ac80ddc854f78bc5cf1b7e8d010cd04263303f51655bbc9"} Nov 03 01:58:50.573183 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:50.573150 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-jqswh" event={"ID":"650460ed-84e8-4f97-9f92-82f5854dd6c7","Type":"ContainerStarted","Data":"e552be443e8f036bc423c809ba7107a4853b5010962b2e106fd13ee727af4fb7"} Nov 03 01:58:50.573607 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:50.573299 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-jqswh" Nov 03 01:58:50.688223 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:50.688144 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-jqswh" podStartSLOduration=2.291157901 podStartE2EDuration="7.688132476s" totalImagesPullingTime="5.396974575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:58:50.686645896 +0000 UTC m=+412.370755051" watchObservedRunningTime="2026-11-03 01:58:50.688132476 +0000 UTC m=+412.372241632" Nov 03 01:58:53.436287 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:53.436257 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:53.605357 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:53.605324 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:53.827918 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:53.827843 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h"] Nov 03 01:58:53.831442 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:53.831427 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h" Nov 03 01:58:53.865582 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:53.865552 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b5071c8-5a0d-46cd-bd1e-28d5632cf091-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-lsc6h\" (UID: \"7b5071c8-5a0d-46cd-bd1e-28d5632cf091\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h" Nov 03 01:58:53.865728 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:53.865673 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5m2g\" (UniqueName: \"kubernetes.io/projected/7b5071c8-5a0d-46cd-bd1e-28d5632cf091-kube-api-access-g5m2g\") pod \"payload-pre-processing-7b98bb678c-lsc6h\" (UID: \"7b5071c8-5a0d-46cd-bd1e-28d5632cf091\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h" Nov 03 01:58:53.884084 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:53.884054 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h"] Nov 03 01:58:53.932912 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:53.932872 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 01:58:53.933092 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:53.932992 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lxnf7\"" Nov 03 01:58:53.966601 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:53.966549 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5m2g\" (UniqueName: \"kubernetes.io/projected/7b5071c8-5a0d-46cd-bd1e-28d5632cf091-kube-api-access-g5m2g\") pod \"payload-pre-processing-7b98bb678c-lsc6h\" (UID: \"7b5071c8-5a0d-46cd-bd1e-28d5632cf091\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h" Nov 03 01:58:53.966792 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:53.966638 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b5071c8-5a0d-46cd-bd1e-28d5632cf091-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-lsc6h\" (UID: \"7b5071c8-5a0d-46cd-bd1e-28d5632cf091\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h" Nov 03 01:58:53.967414 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:53.967393 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b5071c8-5a0d-46cd-bd1e-28d5632cf091-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-lsc6h\" (UID: \"7b5071c8-5a0d-46cd-bd1e-28d5632cf091\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h" Nov 03 01:58:54.023515 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:54.023480 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5m2g\" (UniqueName: \"kubernetes.io/projected/7b5071c8-5a0d-46cd-bd1e-28d5632cf091-kube-api-access-g5m2g\") pod \"payload-pre-processing-7b98bb678c-lsc6h\" (UID: \"7b5071c8-5a0d-46cd-bd1e-28d5632cf091\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h" Nov 03 01:58:54.144124 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:54.144101 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h" Nov 03 01:58:54.504469 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:58:54.504441 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5071c8_5a0d_46cd_bd1e_28d5632cf091.slice/crio-dd6c19d27e543ccebdaa93659ca3bda992f7985f44b131ff2fc6d9553be6760a WatchSource:0}: Error finding container dd6c19d27e543ccebdaa93659ca3bda992f7985f44b131ff2fc6d9553be6760a: Status 404 returned error can't find the container with id dd6c19d27e543ccebdaa93659ca3bda992f7985f44b131ff2fc6d9553be6760a Nov 03 01:58:54.508825 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:54.508789 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h"] Nov 03 01:58:54.593784 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:54.593757 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h" event={"ID":"7b5071c8-5a0d-46cd-bd1e-28d5632cf091","Type":"ContainerStarted","Data":"dd6c19d27e543ccebdaa93659ca3bda992f7985f44b131ff2fc6d9553be6760a"} Nov 03 01:58:58.613798 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:58.613716 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h" event={"ID":"7b5071c8-5a0d-46cd-bd1e-28d5632cf091","Type":"ContainerStarted","Data":"e7b24e7b96683d15e18170239db563aba36d0784c7ac08457f955355fee48c86"} Nov 03 01:58:58.614148 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:58.613866 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h" Nov 03 01:58:58.729535 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:58:58.729485 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h" podStartSLOduration=1.907372939 podStartE2EDuration="5.729469318s" totalImagesPullingTime="3.822096379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:58:58.728169691 +0000 UTC m=+420.412278845" watchObservedRunningTime="2026-11-03 01:58:58.729469318 +0000 UTC m=+420.413578518" Nov 03 01:59:01.580853 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:01.580819 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-jqswh" Nov 03 01:59:01.933304 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:01.933269 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-74bd599f44-rqdlf"] Nov 03 01:59:01.936901 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:01.936884 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74bd599f44-rqdlf" Nov 03 01:59:01.989171 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:01.989140 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-74bd599f44-rqdlf"] Nov 03 01:59:02.040249 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:02.040221 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9262bb6d-129c-442b-9658-408a64ed30a8-metrics-certs\") pod \"ai-gateway-controller-74bd599f44-rqdlf\" (UID: \"9262bb6d-129c-442b-9658-408a64ed30a8\") " pod="opendatahub/ai-gateway-controller-74bd599f44-rqdlf" Nov 03 01:59:02.040381 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:02.040341 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkwmw\" (UniqueName: \"kubernetes.io/projected/9262bb6d-129c-442b-9658-408a64ed30a8-kube-api-access-lkwmw\") pod \"ai-gateway-controller-74bd599f44-rqdlf\" (UID: \"9262bb6d-129c-442b-9658-408a64ed30a8\") " pod="opendatahub/ai-gateway-controller-74bd599f44-rqdlf" Nov 03 01:59:02.141080 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:02.141053 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkwmw\" (UniqueName: \"kubernetes.io/projected/9262bb6d-129c-442b-9658-408a64ed30a8-kube-api-access-lkwmw\") pod \"ai-gateway-controller-74bd599f44-rqdlf\" (UID: \"9262bb6d-129c-442b-9658-408a64ed30a8\") " pod="opendatahub/ai-gateway-controller-74bd599f44-rqdlf" Nov 03 01:59:02.141252 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:02.141104 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9262bb6d-129c-442b-9658-408a64ed30a8-metrics-certs\") pod \"ai-gateway-controller-74bd599f44-rqdlf\" (UID: \"9262bb6d-129c-442b-9658-408a64ed30a8\") " pod="opendatahub/ai-gateway-controller-74bd599f44-rqdlf" Nov 03 01:59:02.143445 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:02.143422 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9262bb6d-129c-442b-9658-408a64ed30a8-metrics-certs\") pod \"ai-gateway-controller-74bd599f44-rqdlf\" (UID: \"9262bb6d-129c-442b-9658-408a64ed30a8\") " pod="opendatahub/ai-gateway-controller-74bd599f44-rqdlf" Nov 03 01:59:02.198167 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:02.198106 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkwmw\" (UniqueName: \"kubernetes.io/projected/9262bb6d-129c-442b-9658-408a64ed30a8-kube-api-access-lkwmw\") pod \"ai-gateway-controller-74bd599f44-rqdlf\" (UID: \"9262bb6d-129c-442b-9658-408a64ed30a8\") " pod="opendatahub/ai-gateway-controller-74bd599f44-rqdlf" Nov 03 01:59:02.249115 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:02.249089 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74bd599f44-rqdlf" Nov 03 01:59:02.614454 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:59:02.614420 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9262bb6d_129c_442b_9658_408a64ed30a8.slice/crio-b0e064ab5abcca167eddc4de2b50100ab275ba6809f65729e38c4519f61a3c09 WatchSource:0}: Error finding container b0e064ab5abcca167eddc4de2b50100ab275ba6809f65729e38c4519f61a3c09: Status 404 returned error can't find the container with id b0e064ab5abcca167eddc4de2b50100ab275ba6809f65729e38c4519f61a3c09 Nov 03 01:59:02.615719 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:02.615690 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-74bd599f44-rqdlf"] Nov 03 01:59:02.634489 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:02.634461 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74bd599f44-rqdlf" event={"ID":"9262bb6d-129c-442b-9658-408a64ed30a8","Type":"ContainerStarted","Data":"b0e064ab5abcca167eddc4de2b50100ab275ba6809f65729e38c4519f61a3c09"} Nov 03 01:59:03.640766 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:03.640737 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74bd599f44-rqdlf" event={"ID":"9262bb6d-129c-442b-9658-408a64ed30a8","Type":"ContainerStarted","Data":"758bc056ab38c4edad710d715d8e9a3cb0f8c976616546e41db0c2fdcd4fc617"} Nov 03 01:59:03.641117 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:03.640841 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-74bd599f44-rqdlf" Nov 03 01:59:03.755418 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:03.755370 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-74bd599f44-rqdlf" podStartSLOduration=1.853022296 podStartE2EDuration="2.755355303s" totalImagesPullingTime="902.333007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:59:03.753488958 +0000 UTC m=+425.437598117" watchObservedRunningTime="2026-11-03 01:59:03.755355303 +0000 UTC m=+425.439464459" Nov 03 01:59:05.787263 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:05.787216 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:09.621734 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:09.621710 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h" Nov 03 01:59:13.005604 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:59:13.005568 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 01:59:14.647811 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:14.647779 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-74bd599f44-rqdlf" Nov 03 01:59:14.813042 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:14.813002 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-jqswh"] Nov 03 01:59:14.813280 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:14.813256 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-jqswh" podUID="650460ed-84e8-4f97-9f92-82f5854dd6c7" containerName="manager" containerID="cri-o://e552be443e8f036bc423c809ba7107a4853b5010962b2e106fd13ee727af4fb7" gracePeriod=30 Nov 03 01:59:15.045591 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:15.045570 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-jqswh" Nov 03 01:59:15.154988 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:15.154958 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/650460ed-84e8-4f97-9f92-82f5854dd6c7-kube-api-access-78khm\" (UniqueName: \"kubernetes.io/projected/650460ed-84e8-4f97-9f92-82f5854dd6c7-kube-api-access-78khm\") pod \"650460ed-84e8-4f97-9f92-82f5854dd6c7\" (UID: \"650460ed-84e8-4f97-9f92-82f5854dd6c7\") " Nov 03 01:59:15.155156 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:15.155010 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/650460ed-84e8-4f97-9f92-82f5854dd6c7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/650460ed-84e8-4f97-9f92-82f5854dd6c7-metrics-certs\") pod \"650460ed-84e8-4f97-9f92-82f5854dd6c7\" (UID: \"650460ed-84e8-4f97-9f92-82f5854dd6c7\") " Nov 03 01:59:15.157495 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:15.157450 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/650460ed-84e8-4f97-9f92-82f5854dd6c7-metrics-certs" pod "650460ed-84e8-4f97-9f92-82f5854dd6c7" (UID: "650460ed-84e8-4f97-9f92-82f5854dd6c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:59:15.157604 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:15.157504 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/650460ed-84e8-4f97-9f92-82f5854dd6c7-kube-api-access-78khm" pod "650460ed-84e8-4f97-9f92-82f5854dd6c7" (UID: "650460ed-84e8-4f97-9f92-82f5854dd6c7"). InnerVolumeSpecName "kube-api-access-78khm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:59:15.256471 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:15.256406 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78khm\" (UniqueName: \"kubernetes.io/projected/650460ed-84e8-4f97-9f92-82f5854dd6c7-kube-api-access-78khm\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:15.256471 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:15.256432 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/650460ed-84e8-4f97-9f92-82f5854dd6c7-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:15.698822 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:15.698790 5463 generic.go:356] "Generic (PLEG): container finished" podID="650460ed-84e8-4f97-9f92-82f5854dd6c7" containerID="e552be443e8f036bc423c809ba7107a4853b5010962b2e106fd13ee727af4fb7" exitCode=0 Nov 03 01:59:15.699236 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:15.698835 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-jqswh" Nov 03 01:59:15.699236 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:15.698872 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-jqswh" event={"ID":"650460ed-84e8-4f97-9f92-82f5854dd6c7","Type":"ContainerDied","Data":"e552be443e8f036bc423c809ba7107a4853b5010962b2e106fd13ee727af4fb7"} Nov 03 01:59:15.699236 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:15.698925 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-jqswh" event={"ID":"650460ed-84e8-4f97-9f92-82f5854dd6c7","Type":"ContainerDied","Data":"69dfce93324250796ac80ddc854f78bc5cf1b7e8d010cd04263303f51655bbc9"} Nov 03 01:59:15.699236 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:15.698947 5463 scope.go:122] "RemoveContainer" containerID="e552be443e8f036bc423c809ba7107a4853b5010962b2e106fd13ee727af4fb7" Nov 03 01:59:15.709603 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:15.709525 5463 scope.go:122] "RemoveContainer" containerID="e552be443e8f036bc423c809ba7107a4853b5010962b2e106fd13ee727af4fb7" Nov 03 01:59:15.710036 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:59:15.709959 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e552be443e8f036bc423c809ba7107a4853b5010962b2e106fd13ee727af4fb7\": container with ID starting with e552be443e8f036bc423c809ba7107a4853b5010962b2e106fd13ee727af4fb7 not found: ID does not exist" containerID="e552be443e8f036bc423c809ba7107a4853b5010962b2e106fd13ee727af4fb7" Nov 03 01:59:15.710036 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:15.709993 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e552be443e8f036bc423c809ba7107a4853b5010962b2e106fd13ee727af4fb7"} err="failed to get container status \"e552be443e8f036bc423c809ba7107a4853b5010962b2e106fd13ee727af4fb7\": rpc error: code = NotFound desc = could not find container \"e552be443e8f036bc423c809ba7107a4853b5010962b2e106fd13ee727af4fb7\": container with ID starting with e552be443e8f036bc423c809ba7107a4853b5010962b2e106fd13ee727af4fb7 not found: ID does not exist" Nov 03 01:59:15.862918 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:15.862890 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-jqswh"] Nov 03 01:59:15.913102 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:15.913073 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-jqswh"] Nov 03 01:59:16.440473 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:16.440442 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="650460ed-84e8-4f97-9f92-82f5854dd6c7" path="/var/lib/kubelet/pods/650460ed-84e8-4f97-9f92-82f5854dd6c7/volumes" Nov 03 01:59:16.563376 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:16.563345 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:17.100743 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:17.100712 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:17.531786 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:17.531754 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:18.391440 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:18.391406 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:18.560307 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:18.560273 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:22.173408 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:22.173376 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:30.035848 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:30.035816 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:35.568303 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.568267 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x"] Nov 03 01:59:35.568707 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.568651 5463 cpu_manager.go:414] "containerMap: removing container" podUID="650460ed-84e8-4f97-9f92-82f5854dd6c7" containerName="manager" Nov 03 01:59:35.568707 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.568661 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="650460ed-84e8-4f97-9f92-82f5854dd6c7" containerName="manager" Nov 03 01:59:35.568798 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.568716 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="650460ed-84e8-4f97-9f92-82f5854dd6c7" containerName="manager" Nov 03 01:59:35.578777 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.578754 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:35.625485 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.625453 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x"] Nov 03 01:59:35.673697 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.673668 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 01:59:35.673838 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.673745 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 01:59:35.673838 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.673754 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nlcr8\"" Nov 03 01:59:35.681687 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.681633 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 01:59:35.681904 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.681883 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:35.729115 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.729085 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de67a6d-f6d3-444b-adc2-743bffd78030-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-66z9x\" (UID: \"9de67a6d-f6d3-444b-adc2-743bffd78030\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:35.729284 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.729123 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9de67a6d-f6d3-444b-adc2-743bffd78030-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-66z9x\" (UID: \"9de67a6d-f6d3-444b-adc2-743bffd78030\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:35.729327 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.729286 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de67a6d-f6d3-444b-adc2-743bffd78030-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-66z9x\" (UID: \"9de67a6d-f6d3-444b-adc2-743bffd78030\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:35.729327 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.729311 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de67a6d-f6d3-444b-adc2-743bffd78030-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-66z9x\" (UID: \"9de67a6d-f6d3-444b-adc2-743bffd78030\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:35.729392 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.729338 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de67a6d-f6d3-444b-adc2-743bffd78030-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-66z9x\" (UID: \"9de67a6d-f6d3-444b-adc2-743bffd78030\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:35.729392 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.729364 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klc7n\" (UniqueName: \"kubernetes.io/projected/9de67a6d-f6d3-444b-adc2-743bffd78030-kube-api-access-klc7n\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-66z9x\" (UID: \"9de67a6d-f6d3-444b-adc2-743bffd78030\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:35.830461 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.830390 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de67a6d-f6d3-444b-adc2-743bffd78030-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-66z9x\" (UID: \"9de67a6d-f6d3-444b-adc2-743bffd78030\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:35.830461 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.830425 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9de67a6d-f6d3-444b-adc2-743bffd78030-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-66z9x\" (UID: \"9de67a6d-f6d3-444b-adc2-743bffd78030\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:35.830461 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.830459 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de67a6d-f6d3-444b-adc2-743bffd78030-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-66z9x\" (UID: \"9de67a6d-f6d3-444b-adc2-743bffd78030\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:35.830712 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.830475 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de67a6d-f6d3-444b-adc2-743bffd78030-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-66z9x\" (UID: \"9de67a6d-f6d3-444b-adc2-743bffd78030\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:35.830712 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.830503 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de67a6d-f6d3-444b-adc2-743bffd78030-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-66z9x\" (UID: \"9de67a6d-f6d3-444b-adc2-743bffd78030\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:35.830712 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.830521 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klc7n\" (UniqueName: \"kubernetes.io/projected/9de67a6d-f6d3-444b-adc2-743bffd78030-kube-api-access-klc7n\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-66z9x\" (UID: \"9de67a6d-f6d3-444b-adc2-743bffd78030\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:35.830880 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.830846 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de67a6d-f6d3-444b-adc2-743bffd78030-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-66z9x\" (UID: \"9de67a6d-f6d3-444b-adc2-743bffd78030\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:35.830939 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.830893 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9de67a6d-f6d3-444b-adc2-743bffd78030-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-66z9x\" (UID: \"9de67a6d-f6d3-444b-adc2-743bffd78030\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:35.830997 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.830965 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de67a6d-f6d3-444b-adc2-743bffd78030-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-66z9x\" (UID: \"9de67a6d-f6d3-444b-adc2-743bffd78030\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:35.833018 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.832992 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de67a6d-f6d3-444b-adc2-743bffd78030-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-66z9x\" (UID: \"9de67a6d-f6d3-444b-adc2-743bffd78030\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:35.833265 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.833247 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de67a6d-f6d3-444b-adc2-743bffd78030-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-66z9x\" (UID: \"9de67a6d-f6d3-444b-adc2-743bffd78030\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:35.886985 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.886955 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klc7n\" (UniqueName: \"kubernetes.io/projected/9de67a6d-f6d3-444b-adc2-743bffd78030-kube-api-access-klc7n\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-66z9x\" (UID: \"9de67a6d-f6d3-444b-adc2-743bffd78030\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:35.890872 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:35.890850 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:36.254702 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:59:36.254669 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de67a6d_f6d3_444b_adc2_743bffd78030.slice/crio-cbaad8fc6e6f2865420edc6bc153b5e592b12b9b4601ceea499a470cdaa6801c WatchSource:0}: Error finding container cbaad8fc6e6f2865420edc6bc153b5e592b12b9b4601ceea499a470cdaa6801c: Status 404 returned error can't find the container with id cbaad8fc6e6f2865420edc6bc153b5e592b12b9b4601ceea499a470cdaa6801c Nov 03 01:59:36.255659 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:36.255633 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x"] Nov 03 01:59:36.804491 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:36.804456 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" event={"ID":"9de67a6d-f6d3-444b-adc2-743bffd78030","Type":"ContainerStarted","Data":"cbaad8fc6e6f2865420edc6bc153b5e592b12b9b4601ceea499a470cdaa6801c"} Nov 03 01:59:37.811472 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:37.811444 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" event={"ID":"9de67a6d-f6d3-444b-adc2-743bffd78030","Type":"ContainerStarted","Data":"8b9122f9c9e2951945fa6f5368d5a6b2468ec76c2d4b73de9581035dbae40b79"} Nov 03 01:59:37.891028 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:37.890998 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:37.925442 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:37.925397 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" podStartSLOduration=1.448764258 podStartE2EDuration="2.92538356s" totalImagesPullingTime="1.476619302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:59:37.925016274 +0000 UTC m=+459.609125429" watchObservedRunningTime="2026-11-03 01:59:37.92538356 +0000 UTC m=+459.609492714" Nov 03 01:59:40.181543 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:40.181443 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:43.007108 ip-10-0-137-69 kubenswrapper[5463]: E1103 01:59:43.007069 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 01:59:43.200866 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.200831 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr"] Nov 03 01:59:43.204081 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.204061 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:43.260055 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.259982 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr"] Nov 03 01:59:43.290771 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.290730 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb32ec19-806e-4490-952f-8134c1a705c1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nptdr\" (UID: \"fb32ec19-806e-4490-952f-8134c1a705c1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:43.290949 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.290787 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb32ec19-806e-4490-952f-8134c1a705c1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nptdr\" (UID: \"fb32ec19-806e-4490-952f-8134c1a705c1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:43.290949 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.290913 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl7mx\" (UniqueName: \"kubernetes.io/projected/fb32ec19-806e-4490-952f-8134c1a705c1-kube-api-access-kl7mx\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nptdr\" (UID: \"fb32ec19-806e-4490-952f-8134c1a705c1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:43.291064 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.290956 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb32ec19-806e-4490-952f-8134c1a705c1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nptdr\" (UID: \"fb32ec19-806e-4490-952f-8134c1a705c1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:43.291064 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.291007 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb32ec19-806e-4490-952f-8134c1a705c1-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nptdr\" (UID: \"fb32ec19-806e-4490-952f-8134c1a705c1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:43.291139 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.291062 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb32ec19-806e-4490-952f-8134c1a705c1-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nptdr\" (UID: \"fb32ec19-806e-4490-952f-8134c1a705c1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:43.305332 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.305302 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:43.307421 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.307405 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 01:59:43.392238 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.392171 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb32ec19-806e-4490-952f-8134c1a705c1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nptdr\" (UID: \"fb32ec19-806e-4490-952f-8134c1a705c1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:43.392418 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.392292 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb32ec19-806e-4490-952f-8134c1a705c1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nptdr\" (UID: \"fb32ec19-806e-4490-952f-8134c1a705c1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:43.392418 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.392356 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl7mx\" (UniqueName: \"kubernetes.io/projected/fb32ec19-806e-4490-952f-8134c1a705c1-kube-api-access-kl7mx\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nptdr\" (UID: \"fb32ec19-806e-4490-952f-8134c1a705c1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:43.392418 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.392385 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb32ec19-806e-4490-952f-8134c1a705c1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nptdr\" (UID: \"fb32ec19-806e-4490-952f-8134c1a705c1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:43.392596 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.392437 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb32ec19-806e-4490-952f-8134c1a705c1-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nptdr\" (UID: \"fb32ec19-806e-4490-952f-8134c1a705c1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:43.392596 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.392477 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb32ec19-806e-4490-952f-8134c1a705c1-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nptdr\" (UID: \"fb32ec19-806e-4490-952f-8134c1a705c1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:43.392720 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.392689 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb32ec19-806e-4490-952f-8134c1a705c1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nptdr\" (UID: \"fb32ec19-806e-4490-952f-8134c1a705c1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:43.392798 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.392779 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb32ec19-806e-4490-952f-8134c1a705c1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nptdr\" (UID: \"fb32ec19-806e-4490-952f-8134c1a705c1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:43.392847 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.392829 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb32ec19-806e-4490-952f-8134c1a705c1-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nptdr\" (UID: \"fb32ec19-806e-4490-952f-8134c1a705c1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:43.395104 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.395072 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb32ec19-806e-4490-952f-8134c1a705c1-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nptdr\" (UID: \"fb32ec19-806e-4490-952f-8134c1a705c1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:43.395342 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.395322 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb32ec19-806e-4490-952f-8134c1a705c1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nptdr\" (UID: \"fb32ec19-806e-4490-952f-8134c1a705c1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:43.449396 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.449365 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl7mx\" (UniqueName: \"kubernetes.io/projected/fb32ec19-806e-4490-952f-8134c1a705c1-kube-api-access-kl7mx\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nptdr\" (UID: \"fb32ec19-806e-4490-952f-8134c1a705c1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:43.519040 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.518936 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:43.944484 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:43.944401 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr"] Nov 03 01:59:43.947434 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:59:43.947389 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb32ec19_806e_4490_952f_8134c1a705c1.slice/crio-e22b003beb07226a200bbad541690a00bd0c6a5520db973926ec26d3b7f9f657 WatchSource:0}: Error finding container e22b003beb07226a200bbad541690a00bd0c6a5520db973926ec26d3b7f9f657: Status 404 returned error can't find the container with id e22b003beb07226a200bbad541690a00bd0c6a5520db973926ec26d3b7f9f657 Nov 03 01:59:44.850290 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:44.850185 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" event={"ID":"fb32ec19-806e-4490-952f-8134c1a705c1","Type":"ContainerStarted","Data":"ffd44d785f07fe944561e0cd3dd5f77962fe670ed01ab4706758d17b28ed01df"} Nov 03 01:59:44.850290 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:44.850272 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" event={"ID":"fb32ec19-806e-4490-952f-8134c1a705c1","Type":"ContainerStarted","Data":"e22b003beb07226a200bbad541690a00bd0c6a5520db973926ec26d3b7f9f657"} Nov 03 01:59:44.964602 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:44.964552 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" podStartSLOduration=1.314730343 podStartE2EDuration="1.96453618s" totalImagesPullingTime="649.805837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:59:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:59:44.964078416 +0000 UTC m=+466.648187571" watchObservedRunningTime="2026-11-03 01:59:44.96453618 +0000 UTC m=+466.648645335" Nov 03 01:59:45.519620 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:45.519589 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:45.891249 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:45.891156 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:45.903838 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:45.903814 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:46.870322 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:46.870290 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-66z9x" Nov 03 01:59:49.060119 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.060086 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs"] Nov 03 01:59:49.063630 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.063615 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:49.117699 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.117666 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs"] Nov 03 01:59:49.148882 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.148851 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad09cdb-425b-4124-8470-1ab16e4719ce-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs\" (UID: \"7ad09cdb-425b-4124-8470-1ab16e4719ce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:49.149099 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.148894 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sjrk\" (UniqueName: \"kubernetes.io/projected/7ad09cdb-425b-4124-8470-1ab16e4719ce-kube-api-access-5sjrk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs\" (UID: \"7ad09cdb-425b-4124-8470-1ab16e4719ce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:49.149099 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.149078 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad09cdb-425b-4124-8470-1ab16e4719ce-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs\" (UID: \"7ad09cdb-425b-4124-8470-1ab16e4719ce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:49.149264 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.149106 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad09cdb-425b-4124-8470-1ab16e4719ce-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs\" (UID: \"7ad09cdb-425b-4124-8470-1ab16e4719ce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:49.149264 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.149130 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad09cdb-425b-4124-8470-1ab16e4719ce-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs\" (UID: \"7ad09cdb-425b-4124-8470-1ab16e4719ce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:49.149374 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.149270 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad09cdb-425b-4124-8470-1ab16e4719ce-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs\" (UID: \"7ad09cdb-425b-4124-8470-1ab16e4719ce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:49.165440 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.165419 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 01:59:49.185298 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.185274 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:49.250689 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.250659 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad09cdb-425b-4124-8470-1ab16e4719ce-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs\" (UID: \"7ad09cdb-425b-4124-8470-1ab16e4719ce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:49.250825 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.250709 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad09cdb-425b-4124-8470-1ab16e4719ce-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs\" (UID: \"7ad09cdb-425b-4124-8470-1ab16e4719ce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:49.250825 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.250742 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sjrk\" (UniqueName: \"kubernetes.io/projected/7ad09cdb-425b-4124-8470-1ab16e4719ce-kube-api-access-5sjrk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs\" (UID: \"7ad09cdb-425b-4124-8470-1ab16e4719ce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:49.250946 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.250848 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad09cdb-425b-4124-8470-1ab16e4719ce-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs\" (UID: \"7ad09cdb-425b-4124-8470-1ab16e4719ce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:49.250946 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.250875 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad09cdb-425b-4124-8470-1ab16e4719ce-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs\" (UID: \"7ad09cdb-425b-4124-8470-1ab16e4719ce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:49.250946 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.250899 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad09cdb-425b-4124-8470-1ab16e4719ce-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs\" (UID: \"7ad09cdb-425b-4124-8470-1ab16e4719ce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:49.251218 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.251162 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad09cdb-425b-4124-8470-1ab16e4719ce-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs\" (UID: \"7ad09cdb-425b-4124-8470-1ab16e4719ce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:49.251377 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.251251 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad09cdb-425b-4124-8470-1ab16e4719ce-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs\" (UID: \"7ad09cdb-425b-4124-8470-1ab16e4719ce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:49.251439 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.251399 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad09cdb-425b-4124-8470-1ab16e4719ce-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs\" (UID: \"7ad09cdb-425b-4124-8470-1ab16e4719ce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:49.253434 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.253411 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad09cdb-425b-4124-8470-1ab16e4719ce-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs\" (UID: \"7ad09cdb-425b-4124-8470-1ab16e4719ce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:49.253814 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.253793 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad09cdb-425b-4124-8470-1ab16e4719ce-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs\" (UID: \"7ad09cdb-425b-4124-8470-1ab16e4719ce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:49.308300 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.308272 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sjrk\" (UniqueName: \"kubernetes.io/projected/7ad09cdb-425b-4124-8470-1ab16e4719ce-kube-api-access-5sjrk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs\" (UID: \"7ad09cdb-425b-4124-8470-1ab16e4719ce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:49.375942 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.375881 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:49.739641 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:59:49.739610 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad09cdb_425b_4124_8470_1ab16e4719ce.slice/crio-5372253f973a1b9e826c4a5f364c3cccb213f6e20ac6b6ade271528603b117bf WatchSource:0}: Error finding container 5372253f973a1b9e826c4a5f364c3cccb213f6e20ac6b6ade271528603b117bf: Status 404 returned error can't find the container with id 5372253f973a1b9e826c4a5f364c3cccb213f6e20ac6b6ade271528603b117bf Nov 03 01:59:49.741620 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.741597 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs"] Nov 03 01:59:49.877167 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:49.877137 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" event={"ID":"7ad09cdb-425b-4124-8470-1ab16e4719ce","Type":"ContainerStarted","Data":"5372253f973a1b9e826c4a5f364c3cccb213f6e20ac6b6ade271528603b117bf"} Nov 03 01:59:50.884486 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:50.884446 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" event={"ID":"7ad09cdb-425b-4124-8470-1ab16e4719ce","Type":"ContainerStarted","Data":"ebc356e1a87db1d904f0ea821641a415342c7a77a8e629c73b8a1896436ec047"} Nov 03 01:59:51.000051 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:50.999996 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" podStartSLOduration=1.390337329 podStartE2EDuration="1.999984189s" totalImagesPullingTime="609.64686ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:59:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:59:50.998613393 +0000 UTC m=+472.682722547" watchObservedRunningTime="2026-11-03 01:59:50.999984189 +0000 UTC m=+472.684093384" Nov 03 01:59:51.376281 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:51.376245 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:53.119365 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:53.119328 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:53.519075 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:53.519040 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:53.531473 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:53.531446 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:53.912674 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:53.912642 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nptdr" Nov 03 01:59:56.628763 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.628731 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q"] Nov 03 01:59:56.634704 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.634115 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:56.687481 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.687448 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q"] Nov 03 01:59:56.722244 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.722209 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3840e493-481d-498a-9ca2-67115c2ec319-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c727q\" (UID: \"3840e493-481d-498a-9ca2-67115c2ec319\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:56.722404 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.722251 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3840e493-481d-498a-9ca2-67115c2ec319-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c727q\" (UID: \"3840e493-481d-498a-9ca2-67115c2ec319\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:56.722404 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.722277 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3840e493-481d-498a-9ca2-67115c2ec319-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c727q\" (UID: \"3840e493-481d-498a-9ca2-67115c2ec319\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:56.722497 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.722417 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3840e493-481d-498a-9ca2-67115c2ec319-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c727q\" (UID: \"3840e493-481d-498a-9ca2-67115c2ec319\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:56.722497 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.722459 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3840e493-481d-498a-9ca2-67115c2ec319-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c727q\" (UID: \"3840e493-481d-498a-9ca2-67115c2ec319\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:56.722497 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.722477 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ldws\" (UniqueName: \"kubernetes.io/projected/3840e493-481d-498a-9ca2-67115c2ec319-kube-api-access-7ldws\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c727q\" (UID: \"3840e493-481d-498a-9ca2-67115c2ec319\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:56.735119 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.735088 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 01:59:56.747886 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.747863 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:56.823787 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.823749 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3840e493-481d-498a-9ca2-67115c2ec319-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c727q\" (UID: \"3840e493-481d-498a-9ca2-67115c2ec319\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:56.823787 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.823790 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3840e493-481d-498a-9ca2-67115c2ec319-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c727q\" (UID: \"3840e493-481d-498a-9ca2-67115c2ec319\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:56.824005 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.823816 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3840e493-481d-498a-9ca2-67115c2ec319-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c727q\" (UID: \"3840e493-481d-498a-9ca2-67115c2ec319\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:56.824005 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.823858 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3840e493-481d-498a-9ca2-67115c2ec319-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c727q\" (UID: \"3840e493-481d-498a-9ca2-67115c2ec319\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:56.824005 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.823897 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3840e493-481d-498a-9ca2-67115c2ec319-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c727q\" (UID: \"3840e493-481d-498a-9ca2-67115c2ec319\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:56.824005 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.823922 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ldws\" (UniqueName: \"kubernetes.io/projected/3840e493-481d-498a-9ca2-67115c2ec319-kube-api-access-7ldws\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c727q\" (UID: \"3840e493-481d-498a-9ca2-67115c2ec319\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:56.824248 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.824222 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3840e493-481d-498a-9ca2-67115c2ec319-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c727q\" (UID: \"3840e493-481d-498a-9ca2-67115c2ec319\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:56.824431 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.824267 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3840e493-481d-498a-9ca2-67115c2ec319-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c727q\" (UID: \"3840e493-481d-498a-9ca2-67115c2ec319\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:56.824660 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.824634 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3840e493-481d-498a-9ca2-67115c2ec319-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c727q\" (UID: \"3840e493-481d-498a-9ca2-67115c2ec319\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:56.826292 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.826267 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3840e493-481d-498a-9ca2-67115c2ec319-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c727q\" (UID: \"3840e493-481d-498a-9ca2-67115c2ec319\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:56.826622 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.826597 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3840e493-481d-498a-9ca2-67115c2ec319-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c727q\" (UID: \"3840e493-481d-498a-9ca2-67115c2ec319\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:56.882608 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.882540 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ldws\" (UniqueName: \"kubernetes.io/projected/3840e493-481d-498a-9ca2-67115c2ec319-kube-api-access-7ldws\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c727q\" (UID: \"3840e493-481d-498a-9ca2-67115c2ec319\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:56.947644 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:56.947542 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:57.320086 ip-10-0-137-69 kubenswrapper[5463]: W1103 01:59:57.320038 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3840e493_481d_498a_9ca2_67115c2ec319.slice/crio-30c964f751bc976e5cb5f435afcd3a27dd0b49d785dbad40c13b0a5be816b710 WatchSource:0}: Error finding container 30c964f751bc976e5cb5f435afcd3a27dd0b49d785dbad40c13b0a5be816b710: Status 404 returned error can't find the container with id 30c964f751bc976e5cb5f435afcd3a27dd0b49d785dbad40c13b0a5be816b710 Nov 03 01:59:57.321389 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:57.321361 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q"] Nov 03 01:59:57.927720 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:57.927692 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" event={"ID":"3840e493-481d-498a-9ca2-67115c2ec319","Type":"ContainerStarted","Data":"30c964f751bc976e5cb5f435afcd3a27dd0b49d785dbad40c13b0a5be816b710"} Nov 03 01:59:58.934059 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:58.934017 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" event={"ID":"3840e493-481d-498a-9ca2-67115c2ec319","Type":"ContainerStarted","Data":"84e66965bbd32471915a226bde102e500cebb9951dcac93950306a54d01fb8f6"} Nov 03 01:59:58.948487 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:58.948465 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 01:59:59.053170 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:59.053117 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" podStartSLOduration=2.452766194 podStartE2EDuration="3.053102302s" totalImagesPullingTime="600.336108ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:59:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:59:59.051480983 +0000 UTC m=+480.735590145" watchObservedRunningTime="2026-11-03 01:59:59.053102302 +0000 UTC m=+480.737211457" Nov 03 01:59:59.376610 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:59.376536 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:59.389143 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:59.389119 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 01:59:59.586259 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:59.586214 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:59.722268 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:59.722234 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:59.902843 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:59.902811 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:59.953097 ip-10-0-137-69 kubenswrapper[5463]: I1103 01:59:59.953068 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs" Nov 03 02:00:00.164456 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:00.164420 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244"] Nov 03 02:00:00.169522 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:00.169503 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244" Nov 03 02:00:00.222074 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:00.222043 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244"] Nov 03 02:00:00.230845 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:00.230822 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:00:00.231084 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:00.231066 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:00:00.231154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:00.231112 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:00:00.257283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:00.257254 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd427\" (UniqueName: \"kubernetes.io/projected/8d77370c-2903-4dba-accb-03ee191220c4-kube-api-access-gd427\") pod \"maas-api-key-cleanup-29894520-kv244\" (UID: \"8d77370c-2903-4dba-accb-03ee191220c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244" Nov 03 02:00:00.257436 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:00.257417 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d77370c-2903-4dba-accb-03ee191220c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894520-kv244\" (UID: \"8d77370c-2903-4dba-accb-03ee191220c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244" Nov 03 02:00:00.358835 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:00.358797 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d77370c-2903-4dba-accb-03ee191220c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894520-kv244\" (UID: \"8d77370c-2903-4dba-accb-03ee191220c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244" Nov 03 02:00:00.359019 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:00.358866 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd427\" (UniqueName: \"kubernetes.io/projected/8d77370c-2903-4dba-accb-03ee191220c4-kube-api-access-gd427\") pod \"maas-api-key-cleanup-29894520-kv244\" (UID: \"8d77370c-2903-4dba-accb-03ee191220c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244" Nov 03 02:00:00.359525 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:00.359502 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d77370c-2903-4dba-accb-03ee191220c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894520-kv244\" (UID: \"8d77370c-2903-4dba-accb-03ee191220c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244" Nov 03 02:00:00.416128 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:00.416048 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd427\" (UniqueName: \"kubernetes.io/projected/8d77370c-2903-4dba-accb-03ee191220c4-kube-api-access-gd427\") pod \"maas-api-key-cleanup-29894520-kv244\" (UID: \"8d77370c-2903-4dba-accb-03ee191220c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244" Nov 03 02:00:00.485496 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:00.485467 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244" Nov 03 02:00:00.847486 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:00:00.847445 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d77370c_2903_4dba_accb_03ee191220c4.slice/crio-21a1205961b00fc097356756a3d3997deb04c5305df12081012051a9d4984ddf WatchSource:0}: Error finding container 21a1205961b00fc097356756a3d3997deb04c5305df12081012051a9d4984ddf: Status 404 returned error can't find the container with id 21a1205961b00fc097356756a3d3997deb04c5305df12081012051a9d4984ddf Nov 03 02:00:00.849332 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:00.849288 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244"] Nov 03 02:00:00.948683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:00.948639 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244" event={"ID":"8d77370c-2903-4dba-accb-03ee191220c4","Type":"ContainerStarted","Data":"21a1205961b00fc097356756a3d3997deb04c5305df12081012051a9d4984ddf"} Nov 03 02:00:06.118253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:06.118218 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:06.948615 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:06.948582 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 02:00:06.965370 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:06.965345 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 02:00:06.998925 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:06.998898 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c727q" Nov 03 02:00:08.997089 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:08.997049 5463 generic.go:356] "Generic (PLEG): container finished" podID="8d77370c-2903-4dba-accb-03ee191220c4" containerID="a8969fd86c3a72b29dbdeb7015e78cc1c05d3cea89d3ade7d14c6a6e9064ad47" exitCode=60 Nov 03 02:00:08.997556 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:08.997132 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244" event={"ID":"8d77370c-2903-4dba-accb-03ee191220c4","Type":"ContainerDied","Data":"a8969fd86c3a72b29dbdeb7015e78cc1c05d3cea89d3ade7d14c6a6e9064ad47"} Nov 03 02:00:08.997556 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:08.997380 5463 scope.go:122] "RemoveContainer" containerID="a8969fd86c3a72b29dbdeb7015e78cc1c05d3cea89d3ade7d14c6a6e9064ad47" Nov 03 02:00:10.003225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:10.003127 5463 generic.go:356] "Generic (PLEG): container finished" podID="8d77370c-2903-4dba-accb-03ee191220c4" containerID="b0da59912a835dc44350d844c1c7c55cefda006012314f4b73f4352043e124c1" exitCode=60 Nov 03 02:00:10.003225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:10.003215 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244" event={"ID":"8d77370c-2903-4dba-accb-03ee191220c4","Type":"ContainerDied","Data":"b0da59912a835dc44350d844c1c7c55cefda006012314f4b73f4352043e124c1"} Nov 03 02:00:10.003678 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:10.003269 5463 scope.go:122] "RemoveContainer" containerID="a8969fd86c3a72b29dbdeb7015e78cc1c05d3cea89d3ade7d14c6a6e9064ad47" Nov 03 02:00:10.003678 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:10.003453 5463 scope.go:122] "RemoveContainer" containerID="b0da59912a835dc44350d844c1c7c55cefda006012314f4b73f4352043e124c1" Nov 03 02:00:10.003777 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:00:10.003679 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894520-kv244_odh-ai-gateway-infra(8d77370c-2903-4dba-accb-03ee191220c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244" podUID="8d77370c-2903-4dba-accb-03ee191220c4" Nov 03 02:00:11.011265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:11.011240 5463 scope.go:122] "RemoveContainer" containerID="b0da59912a835dc44350d844c1c7c55cefda006012314f4b73f4352043e124c1" Nov 03 02:00:11.011654 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:00:11.011467 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894520-kv244_odh-ai-gateway-infra(8d77370c-2903-4dba-accb-03ee191220c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244" podUID="8d77370c-2903-4dba-accb-03ee191220c4" Nov 03 02:00:13.007161 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:00:13.007118 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:00:18.598417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.598384 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq"] Nov 03 02:00:18.602939 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.602919 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.710840 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.710810 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq"] Nov 03 02:00:18.737587 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.737560 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab070df-285a-4b9f-8d66-b0b6f976cecb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.737751 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.737592 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.737751 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.737610 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.737833 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.737770 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.737833 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.737798 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whj66\" (UniqueName: \"kubernetes.io/projected/dab070df-285a-4b9f-8d66-b0b6f976cecb-kube-api-access-whj66\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.737833 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.737827 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.737986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.737963 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.738058 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.738034 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.738113 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.738071 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.839505 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.839470 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.839505 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.839505 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.839721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.839533 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab070df-285a-4b9f-8d66-b0b6f976cecb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.839721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.839645 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.839838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.839738 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.839838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.839828 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.839949 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.839848 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whj66\" (UniqueName: \"kubernetes.io/projected/dab070df-285a-4b9f-8d66-b0b6f976cecb-kube-api-access-whj66\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.839949 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.839873 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.840056 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.839970 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.840056 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.839986 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.840154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.840091 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.840281 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.840259 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.840361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.840311 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.840419 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.840394 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab070df-285a-4b9f-8d66-b0b6f976cecb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.842096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.842074 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.842659 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.842641 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.896451 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.896419 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.896832 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.896814 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whj66\" (UniqueName: \"kubernetes.io/projected/dab070df-285a-4b9f-8d66-b0b6f976cecb-kube-api-access-whj66\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9f4sq\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:18.915911 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:18.915891 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:19.327712 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:00:19.327671 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab070df_285a_4b9f_8d66_b0b6f976cecb.slice/crio-a27865e00dd946749c56dab5c3fb151248b8cfe88726f380f721f454b0889719 WatchSource:0}: Error finding container a27865e00dd946749c56dab5c3fb151248b8cfe88726f380f721f454b0889719: Status 404 returned error can't find the container with id a27865e00dd946749c56dab5c3fb151248b8cfe88726f380f721f454b0889719 Nov 03 02:00:19.330003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:19.329980 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq"] Nov 03 02:00:19.330121 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:19.330091 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:00:19.330166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:19.330154 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:00:19.330218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:19.330182 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:00:20.077960 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:20.077927 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" event={"ID":"dab070df-285a-4b9f-8d66-b0b6f976cecb","Type":"ContainerStarted","Data":"ac3e6c0512defa4a78c01cc97b784c8c39a6ebf0d5c85dbf225070f291e07e06"} Nov 03 02:00:20.078461 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:20.077987 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" event={"ID":"dab070df-285a-4b9f-8d66-b0b6f976cecb","Type":"ContainerStarted","Data":"a27865e00dd946749c56dab5c3fb151248b8cfe88726f380f721f454b0889719"} Nov 03 02:00:20.196650 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:20.196592 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" podStartSLOduration=2.196578317 podStartE2EDuration="2.196578317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:20.196049182 +0000 UTC m=+501.880158328" watchObservedRunningTime="2026-11-03 02:00:20.196578317 +0000 UTC m=+501.880687471" Nov 03 02:00:20.916932 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:20.916898 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:20.918249 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:20.918227 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9f4sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 02:00:20.918341 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:20.918278 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" podUID="dab070df-285a-4b9f-8d66-b0b6f976cecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 02:00:21.436049 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:21.436019 5463 scope.go:122] "RemoveContainer" containerID="b0da59912a835dc44350d844c1c7c55cefda006012314f4b73f4352043e124c1" Nov 03 02:00:21.917091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:21.917063 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9f4sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 02:00:21.917283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:21.917108 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" podUID="dab070df-285a-4b9f-8d66-b0b6f976cecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 02:00:22.089859 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:22.089768 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244" event={"ID":"8d77370c-2903-4dba-accb-03ee191220c4","Type":"ContainerStarted","Data":"b3b4c70d383d702ff1631c9e4ee4ab7ac73e47490b9e841b77d6f914bd4501bf"} Nov 03 02:00:22.205357 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:22.205298 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244" podStartSLOduration=15.038208313 podStartE2EDuration="22.205280317s" totalImagesPullingTime="7.167072004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:22.203775196 +0000 UTC m=+503.887884351" watchObservedRunningTime="2026-11-03 02:00:22.205280317 +0000 UTC m=+503.889389472" Nov 03 02:00:22.917149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:22.917119 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9f4sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 02:00:22.917574 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:22.917168 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" podUID="dab070df-285a-4b9f-8d66-b0b6f976cecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 02:00:23.096465 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:23.096434 5463 generic.go:356] "Generic (PLEG): container finished" podID="8d77370c-2903-4dba-accb-03ee191220c4" containerID="b3b4c70d383d702ff1631c9e4ee4ab7ac73e47490b9e841b77d6f914bd4501bf" exitCode=60 Nov 03 02:00:23.096617 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:23.096510 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244" event={"ID":"8d77370c-2903-4dba-accb-03ee191220c4","Type":"ContainerDied","Data":"b3b4c70d383d702ff1631c9e4ee4ab7ac73e47490b9e841b77d6f914bd4501bf"} Nov 03 02:00:23.096617 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:23.096574 5463 scope.go:122] "RemoveContainer" containerID="b0da59912a835dc44350d844c1c7c55cefda006012314f4b73f4352043e124c1" Nov 03 02:00:23.096880 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:23.096863 5463 scope.go:122] "RemoveContainer" containerID="b3b4c70d383d702ff1631c9e4ee4ab7ac73e47490b9e841b77d6f914bd4501bf" Nov 03 02:00:23.097116 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:00:23.097099 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894520-kv244_odh-ai-gateway-infra(8d77370c-2903-4dba-accb-03ee191220c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244" podUID="8d77370c-2903-4dba-accb-03ee191220c4" Nov 03 02:00:23.220824 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:23.220743 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244"] Nov 03 02:00:23.916591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:23.916558 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9f4sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 02:00:23.916771 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:23.916617 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" podUID="dab070df-285a-4b9f-8d66-b0b6f976cecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 02:00:24.283318 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:24.283287 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244" Nov 03 02:00:24.305919 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:24.305886 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d77370c-2903-4dba-accb-03ee191220c4-kube-api-access-gd427\" (UniqueName: \"kubernetes.io/projected/8d77370c-2903-4dba-accb-03ee191220c4-kube-api-access-gd427\") pod \"8d77370c-2903-4dba-accb-03ee191220c4\" (UID: \"8d77370c-2903-4dba-accb-03ee191220c4\") " Nov 03 02:00:24.306016 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:24.305927 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d77370c-2903-4dba-accb-03ee191220c4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d77370c-2903-4dba-accb-03ee191220c4-maas-api-service-ca\") pod \"8d77370c-2903-4dba-accb-03ee191220c4\" (UID: \"8d77370c-2903-4dba-accb-03ee191220c4\") " Nov 03 02:00:24.306324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:24.306292 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d77370c-2903-4dba-accb-03ee191220c4-maas-api-service-ca" pod "8d77370c-2903-4dba-accb-03ee191220c4" (UID: "8d77370c-2903-4dba-accb-03ee191220c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:00:24.307976 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:24.307957 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d77370c-2903-4dba-accb-03ee191220c4-kube-api-access-gd427" pod "8d77370c-2903-4dba-accb-03ee191220c4" (UID: "8d77370c-2903-4dba-accb-03ee191220c4"). InnerVolumeSpecName "kube-api-access-gd427". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:00:24.406775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:24.406741 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd427\" (UniqueName: \"kubernetes.io/projected/8d77370c-2903-4dba-accb-03ee191220c4-kube-api-access-gd427\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:00:24.406775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:24.406778 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d77370c-2903-4dba-accb-03ee191220c4-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:00:24.916886 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:24.916849 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9f4sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 02:00:24.917099 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:24.916917 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" podUID="dab070df-285a-4b9f-8d66-b0b6f976cecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 02:00:25.109493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.109460 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244" Nov 03 02:00:25.109493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.109486 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244" event={"ID":"8d77370c-2903-4dba-accb-03ee191220c4","Type":"ContainerDied","Data":"21a1205961b00fc097356756a3d3997deb04c5305df12081012051a9d4984ddf"} Nov 03 02:00:25.109706 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.109550 5463 scope.go:122] "RemoveContainer" containerID="b3b4c70d383d702ff1631c9e4ee4ab7ac73e47490b9e841b77d6f914bd4501bf" Nov 03 02:00:25.280304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.280221 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244"] Nov 03 02:00:25.282865 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.282836 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894520-kv244"] Nov 03 02:00:25.751373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.751343 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c4ff9d6d7-pr8r7"] Nov 03 02:00:25.751771 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.751757 5463 cpu_manager.go:414] "containerMap: removing container" podUID="8d77370c-2903-4dba-accb-03ee191220c4" containerName="cleanup" Nov 03 02:00:25.751821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.751772 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d77370c-2903-4dba-accb-03ee191220c4" containerName="cleanup" Nov 03 02:00:25.751821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.751812 5463 cpu_manager.go:414] "containerMap: removing container" podUID="8d77370c-2903-4dba-accb-03ee191220c4" containerName="cleanup" Nov 03 02:00:25.751821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.751817 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d77370c-2903-4dba-accb-03ee191220c4" containerName="cleanup" Nov 03 02:00:25.751910 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.751877 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d77370c-2903-4dba-accb-03ee191220c4" containerName="cleanup" Nov 03 02:00:25.751910 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.751893 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d77370c-2903-4dba-accb-03ee191220c4" containerName="cleanup" Nov 03 02:00:25.755667 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.755650 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c4ff9d6d7-pr8r7" Nov 03 02:00:25.807415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.807384 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c4ff9d6d7-pr8r7"] Nov 03 02:00:25.818299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.818276 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0f91465-9537-4ffb-b2c9-3d74c20ddced-tls-cert\") pod \"authorino-7c4ff9d6d7-pr8r7\" (UID: \"f0f91465-9537-4ffb-b2c9-3d74c20ddced\") " pod="kuadrant-system/authorino-7c4ff9d6d7-pr8r7" Nov 03 02:00:25.818406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.818312 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m7qg\" (UniqueName: \"kubernetes.io/projected/f0f91465-9537-4ffb-b2c9-3d74c20ddced-kube-api-access-6m7qg\") pod \"authorino-7c4ff9d6d7-pr8r7\" (UID: \"f0f91465-9537-4ffb-b2c9-3d74c20ddced\") " pod="kuadrant-system/authorino-7c4ff9d6d7-pr8r7" Nov 03 02:00:25.916454 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.916425 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9f4sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 02:00:25.916635 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.916481 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" podUID="dab070df-285a-4b9f-8d66-b0b6f976cecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 02:00:25.918923 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.918899 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0f91465-9537-4ffb-b2c9-3d74c20ddced-tls-cert\") pod \"authorino-7c4ff9d6d7-pr8r7\" (UID: \"f0f91465-9537-4ffb-b2c9-3d74c20ddced\") " pod="kuadrant-system/authorino-7c4ff9d6d7-pr8r7" Nov 03 02:00:25.919013 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.918940 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m7qg\" (UniqueName: \"kubernetes.io/projected/f0f91465-9537-4ffb-b2c9-3d74c20ddced-kube-api-access-6m7qg\") pod \"authorino-7c4ff9d6d7-pr8r7\" (UID: \"f0f91465-9537-4ffb-b2c9-3d74c20ddced\") " pod="kuadrant-system/authorino-7c4ff9d6d7-pr8r7" Nov 03 02:00:25.921452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.921432 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0f91465-9537-4ffb-b2c9-3d74c20ddced-tls-cert\") pod \"authorino-7c4ff9d6d7-pr8r7\" (UID: \"f0f91465-9537-4ffb-b2c9-3d74c20ddced\") " pod="kuadrant-system/authorino-7c4ff9d6d7-pr8r7" Nov 03 02:00:25.975802 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:25.975773 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m7qg\" (UniqueName: \"kubernetes.io/projected/f0f91465-9537-4ffb-b2c9-3d74c20ddced-kube-api-access-6m7qg\") pod \"authorino-7c4ff9d6d7-pr8r7\" (UID: \"f0f91465-9537-4ffb-b2c9-3d74c20ddced\") " pod="kuadrant-system/authorino-7c4ff9d6d7-pr8r7" Nov 03 02:00:26.068224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:26.068142 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c4ff9d6d7-pr8r7" Nov 03 02:00:26.418882 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:26.418846 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:26.442753 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:26.442727 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d77370c-2903-4dba-accb-03ee191220c4" path="/var/lib/kubelet/pods/8d77370c-2903-4dba-accb-03ee191220c4/volumes" Nov 03 02:00:26.444556 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:00:26.444529 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0f91465_9537_4ffb_b2c9_3d74c20ddced.slice/crio-eb1cc606dfcdaf1911aed40da190d8779b3ed5385a58f1e0a81856f2a3fe0931 WatchSource:0}: Error finding container eb1cc606dfcdaf1911aed40da190d8779b3ed5385a58f1e0a81856f2a3fe0931: Status 404 returned error can't find the container with id eb1cc606dfcdaf1911aed40da190d8779b3ed5385a58f1e0a81856f2a3fe0931 Nov 03 02:00:26.446542 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:26.446519 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c4ff9d6d7-pr8r7"] Nov 03 02:00:26.571556 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:26.571523 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:27.125184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:27.125143 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c4ff9d6d7-pr8r7" event={"ID":"f0f91465-9537-4ffb-b2c9-3d74c20ddced","Type":"ContainerStarted","Data":"eb1cc606dfcdaf1911aed40da190d8779b3ed5385a58f1e0a81856f2a3fe0931"} Nov 03 02:00:27.707177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:27.707149 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:27.707468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:27.707442 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:27.708263 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:27.708239 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:00:28.018578 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:28.018505 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:28.170159 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:28.170131 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:29.136928 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:29.136836 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c4ff9d6d7-pr8r7" event={"ID":"f0f91465-9537-4ffb-b2c9-3d74c20ddced","Type":"ContainerStarted","Data":"50bc21c2a2a3c5fa13f3aa01c8022f3baa832b5b1c917f2147666690d5404ce0"} Nov 03 02:00:29.251531 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:29.251478 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7c4ff9d6d7-pr8r7" podStartSLOduration=2.032531696 podStartE2EDuration="4.2514634s" totalImagesPullingTime="2.218931704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:29.250164651 +0000 UTC m=+510.934273808" watchObservedRunningTime="2026-11-03 02:00:29.2514634 +0000 UTC m=+510.935572554" Nov 03 02:00:29.275824 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:29.275790 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78dd457b-trpxb"] Nov 03 02:00:29.276230 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:29.276028 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-78dd457b-trpxb" podUID="651e5b21-a142-4a71-b744-d8f198e32183" containerName="authorino" containerID="cri-o://a123dc2bb649480a6d7f68e101ced74bcc2c56efffcc732dc3ba0c4daf4716eb" gracePeriod=30 Nov 03 02:00:29.533535 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:29.533510 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78dd457b-trpxb" Nov 03 02:00:29.551405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:29.551380 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651e5b21-a142-4a71-b744-d8f198e32183-kube-api-access-tdxj8\" (UniqueName: \"kubernetes.io/projected/651e5b21-a142-4a71-b744-d8f198e32183-kube-api-access-tdxj8\") pod \"651e5b21-a142-4a71-b744-d8f198e32183\" (UID: \"651e5b21-a142-4a71-b744-d8f198e32183\") " Nov 03 02:00:29.551564 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:29.551424 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/651e5b21-a142-4a71-b744-d8f198e32183-tls-cert\" (UniqueName: \"kubernetes.io/secret/651e5b21-a142-4a71-b744-d8f198e32183-tls-cert\") pod \"651e5b21-a142-4a71-b744-d8f198e32183\" (UID: \"651e5b21-a142-4a71-b744-d8f198e32183\") " Nov 03 02:00:29.553562 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:29.553526 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651e5b21-a142-4a71-b744-d8f198e32183-kube-api-access-tdxj8" pod "651e5b21-a142-4a71-b744-d8f198e32183" (UID: "651e5b21-a142-4a71-b744-d8f198e32183"). InnerVolumeSpecName "kube-api-access-tdxj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:00:29.561764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:29.561734 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/651e5b21-a142-4a71-b744-d8f198e32183-tls-cert" pod "651e5b21-a142-4a71-b744-d8f198e32183" (UID: "651e5b21-a142-4a71-b744-d8f198e32183"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:00:29.652554 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:29.652515 5463 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/651e5b21-a142-4a71-b744-d8f198e32183-tls-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:00:29.652554 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:29.652551 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdxj8\" (UniqueName: \"kubernetes.io/projected/651e5b21-a142-4a71-b744-d8f198e32183-kube-api-access-tdxj8\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:00:30.144379 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:30.144346 5463 generic.go:356] "Generic (PLEG): container finished" podID="651e5b21-a142-4a71-b744-d8f198e32183" containerID="a123dc2bb649480a6d7f68e101ced74bcc2c56efffcc732dc3ba0c4daf4716eb" exitCode=0 Nov 03 02:00:30.144533 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:30.144391 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78dd457b-trpxb" Nov 03 02:00:30.144533 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:30.144432 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78dd457b-trpxb" event={"ID":"651e5b21-a142-4a71-b744-d8f198e32183","Type":"ContainerDied","Data":"a123dc2bb649480a6d7f68e101ced74bcc2c56efffcc732dc3ba0c4daf4716eb"} Nov 03 02:00:30.144533 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:30.144470 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78dd457b-trpxb" event={"ID":"651e5b21-a142-4a71-b744-d8f198e32183","Type":"ContainerDied","Data":"ea5873c20af686291c33d3f35985c4d85876eafa76de688088286f3cbc3e2cb8"} Nov 03 02:00:30.144533 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:30.144490 5463 scope.go:122] "RemoveContainer" containerID="a123dc2bb649480a6d7f68e101ced74bcc2c56efffcc732dc3ba0c4daf4716eb" Nov 03 02:00:30.154594 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:30.154578 5463 scope.go:122] "RemoveContainer" containerID="a123dc2bb649480a6d7f68e101ced74bcc2c56efffcc732dc3ba0c4daf4716eb" Nov 03 02:00:30.154841 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:00:30.154825 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a123dc2bb649480a6d7f68e101ced74bcc2c56efffcc732dc3ba0c4daf4716eb\": container with ID starting with a123dc2bb649480a6d7f68e101ced74bcc2c56efffcc732dc3ba0c4daf4716eb not found: ID does not exist" containerID="a123dc2bb649480a6d7f68e101ced74bcc2c56efffcc732dc3ba0c4daf4716eb" Nov 03 02:00:30.154891 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:30.154848 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a123dc2bb649480a6d7f68e101ced74bcc2c56efffcc732dc3ba0c4daf4716eb"} err="failed to get container status \"a123dc2bb649480a6d7f68e101ced74bcc2c56efffcc732dc3ba0c4daf4716eb\": rpc error: code = NotFound desc = could not find container \"a123dc2bb649480a6d7f68e101ced74bcc2c56efffcc732dc3ba0c4daf4716eb\": container with ID starting with a123dc2bb649480a6d7f68e101ced74bcc2c56efffcc732dc3ba0c4daf4716eb not found: ID does not exist" Nov 03 02:00:30.308373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:30.308321 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78dd457b-trpxb"] Nov 03 02:00:30.311996 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:30.311964 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-78dd457b-trpxb"] Nov 03 02:00:30.450929 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:00:30.450826 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651e5b21-a142-4a71-b744-d8f198e32183" path="/var/lib/kubelet/pods/651e5b21-a142-4a71-b744-d8f198e32183/volumes" Nov 03 02:00:43.005992 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:00:43.005958 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:01:03.072631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.072587 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:03.761523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.761492 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt"] Nov 03 02:01:03.761928 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.761915 5463 cpu_manager.go:414] "containerMap: removing container" podUID="8d77370c-2903-4dba-accb-03ee191220c4" containerName="cleanup" Nov 03 02:01:03.761980 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.761930 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d77370c-2903-4dba-accb-03ee191220c4" containerName="cleanup" Nov 03 02:01:03.761980 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.761943 5463 cpu_manager.go:414] "containerMap: removing container" podUID="651e5b21-a142-4a71-b744-d8f198e32183" containerName="authorino" Nov 03 02:01:03.761980 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.761948 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651e5b21-a142-4a71-b744-d8f198e32183" containerName="authorino" Nov 03 02:01:03.762085 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.762023 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="651e5b21-a142-4a71-b744-d8f198e32183" containerName="authorino" Nov 03 02:01:03.762085 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.762033 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d77370c-2903-4dba-accb-03ee191220c4" containerName="cleanup" Nov 03 02:01:03.767559 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.767533 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.819920 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.819892 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt"] Nov 03 02:01:03.886730 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.886699 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6ac43c49-openshift-default-dockercfg-f8mj6\"" Nov 03 02:01:03.887019 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.886998 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ad5587-9637-45ea-9d12-97ba06c7011d-istiod-ca-cert\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.887062 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.887031 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-podinfo\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.887062 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.887050 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-data\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.887127 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.887104 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-token\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.887231 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.887214 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-credential-socket\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.887283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.887252 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-workload-socket\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.887283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.887270 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-envoy\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.887411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.887390 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-workload-certs\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.887465 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.887451 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r2ld\" (UniqueName: \"kubernetes.io/projected/c8ad5587-9637-45ea-9d12-97ba06c7011d-kube-api-access-9r2ld\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.988492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.988461 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ad5587-9637-45ea-9d12-97ba06c7011d-istiod-ca-cert\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.988492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.988494 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-podinfo\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.988707 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.988511 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-data\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.988707 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.988535 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-token\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.988825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.988704 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-credential-socket\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.988825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.988796 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-workload-socket\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.988944 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.988823 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-envoy\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.988944 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.988898 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-workload-certs\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.989055 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.988960 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r2ld\" (UniqueName: \"kubernetes.io/projected/c8ad5587-9637-45ea-9d12-97ba06c7011d-kube-api-access-9r2ld\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.989055 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.988967 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-data\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.989303 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.989276 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ad5587-9637-45ea-9d12-97ba06c7011d-istiod-ca-cert\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.989449 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.989368 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-credential-socket\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.989741 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.989718 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-workload-certs\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.990105 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.990082 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-workload-socket\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.991485 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.991456 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-envoy\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:03.992818 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:03.992796 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-podinfo\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:04.049583 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.049532 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-token\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:04.050341 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.050322 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r2ld\" (UniqueName: \"kubernetes.io/projected/c8ad5587-9637-45ea-9d12-97ba06c7011d-kube-api-access-9r2ld\") pod \"e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:04.053767 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.053740 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz"] Nov 03 02:01:04.059513 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.059493 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.081535 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.081512 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:04.113641 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.113606 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz"] Nov 03 02:01:04.119000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.118960 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-aead6b17-openshift-default-dockercfg-ndzfl\"" Nov 03 02:01:04.192624 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.192404 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-workload-socket\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.192624 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.192463 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-token\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.192624 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.192599 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-envoy\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.193261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.192655 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-data\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.193261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.192719 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-credential-socket\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.193261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.192756 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istiod-ca-cert\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.193261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.192784 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-workload-certs\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.193261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.192810 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf8sr\" (UniqueName: \"kubernetes.io/projected/cc06054f-00b6-4308-ac4b-a315a36ef0ba-kube-api-access-qf8sr\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.193261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.192830 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-podinfo\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.293938 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.293905 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-workload-socket\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.293938 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.293940 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-token\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.294161 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.294076 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-envoy\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.294161 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.294115 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-data\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.294161 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.294151 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-credential-socket\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.294351 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.294222 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istiod-ca-cert\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.294351 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.294268 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-workload-certs\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.294351 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.294308 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf8sr\" (UniqueName: \"kubernetes.io/projected/cc06054f-00b6-4308-ac4b-a315a36ef0ba-kube-api-access-qf8sr\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.294351 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.294339 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-podinfo\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.294559 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.294401 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-workload-socket\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.294559 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.294517 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-credential-socket\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.294654 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.294566 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-data\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.294694 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.294653 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-workload-certs\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.294919 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.294896 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istiod-ca-cert\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.296225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.296209 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-envoy\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.296621 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.296598 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-podinfo\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.351476 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.351417 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-token\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.351622 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.351606 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf8sr\" (UniqueName: \"kubernetes.io/projected/cc06054f-00b6-4308-ac4b-a315a36ef0ba-kube-api-access-qf8sr\") pod \"e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.376180 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.376146 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:04.501580 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:01:04.501542 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ad5587_9637_45ea_9d12_97ba06c7011d.slice/crio-851d5a54c471135a14121a7a29372e5d86b1a722832ce0a0c20ac1b7f9724733 WatchSource:0}: Error finding container 851d5a54c471135a14121a7a29372e5d86b1a722832ce0a0c20ac1b7f9724733: Status 404 returned error can't find the container with id 851d5a54c471135a14121a7a29372e5d86b1a722832ce0a0c20ac1b7f9724733 Nov 03 02:01:04.503980 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.503954 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt"] Nov 03 02:01:04.504172 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.504141 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:04.504314 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.504253 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:04.504314 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.504299 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:04.598602 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.598561 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:04.617433 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.617362 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n"] Nov 03 02:01:04.621725 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.621706 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.675222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.675135 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-7683f606-gw-openshift-default-dockercfg-gb4s8\"" Nov 03 02:01:04.677432 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.677364 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n"] Nov 03 02:01:04.752161 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:01:04.752128 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc06054f_00b6_4308_ac4b_a315a36ef0ba.slice/crio-9351807f7b87ece46510cf755eb6c11893ade7a947116008d0f743096ddea117 WatchSource:0}: Error finding container 9351807f7b87ece46510cf755eb6c11893ade7a947116008d0f743096ddea117: Status 404 returned error can't find the container with id 9351807f7b87ece46510cf755eb6c11893ade7a947116008d0f743096ddea117 Nov 03 02:01:04.754375 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.754289 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz"] Nov 03 02:01:04.757021 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.755447 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:04.757021 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.755531 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:04.757021 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.755571 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:04.774301 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.774264 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:04.799357 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.799331 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c00682-647d-4422-8ba4-1149c73b3eee-istio-token\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.799453 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.799386 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-istio-data\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.799493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.799453 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-credential-socket\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.799528 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.799490 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-istio-envoy\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.799528 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.799519 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c00682-647d-4422-8ba4-1149c73b3eee-istiod-ca-cert\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.799666 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.799648 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-workload-certs\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.799704 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.799673 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-workload-socket\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.799744 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.799724 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nthhf\" (UniqueName: \"kubernetes.io/projected/02c00682-647d-4422-8ba4-1149c73b3eee-kube-api-access-nthhf\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.799868 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.799845 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c00682-647d-4422-8ba4-1149c73b3eee-istio-podinfo\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.901243 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.901215 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c00682-647d-4422-8ba4-1149c73b3eee-istio-podinfo\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.901442 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.901261 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c00682-647d-4422-8ba4-1149c73b3eee-istio-token\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.901442 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.901302 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-istio-data\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.901442 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.901329 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-credential-socket\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.901442 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.901369 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-istio-envoy\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.901442 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.901394 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c00682-647d-4422-8ba4-1149c73b3eee-istiod-ca-cert\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.901703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.901475 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-workload-certs\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.901703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.901504 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-workload-socket\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.901703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.901584 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nthhf\" (UniqueName: \"kubernetes.io/projected/02c00682-647d-4422-8ba4-1149c73b3eee-kube-api-access-nthhf\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.901840 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.901820 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-credential-socket\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.901896 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.901839 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-istio-data\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.902106 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.902045 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-workload-certs\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.902266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.902110 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-workload-socket\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.902625 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.902525 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c00682-647d-4422-8ba4-1149c73b3eee-istiod-ca-cert\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.904022 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.903991 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-istio-envoy\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.904618 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.904596 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c00682-647d-4422-8ba4-1149c73b3eee-istio-podinfo\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.933329 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.933295 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:04.958873 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.958845 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c00682-647d-4422-8ba4-1149c73b3eee-istio-token\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:04.959023 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:04.959006 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nthhf\" (UniqueName: \"kubernetes.io/projected/02c00682-647d-4422-8ba4-1149c73b3eee-kube-api-access-nthhf\") pod \"e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:05.170173 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:05.170076 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:05.243700 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:05.243671 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:05.333828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:05.333788 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" event={"ID":"cc06054f-00b6-4308-ac4b-a315a36ef0ba","Type":"ContainerStarted","Data":"34a8539a377ca9d6fd6acaa63603a5715158a5be41a13052018289cd55c66fe3"} Nov 03 02:01:05.333999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:05.333884 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" event={"ID":"cc06054f-00b6-4308-ac4b-a315a36ef0ba","Type":"ContainerStarted","Data":"9351807f7b87ece46510cf755eb6c11893ade7a947116008d0f743096ddea117"} Nov 03 02:01:05.335640 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:05.335610 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" event={"ID":"c8ad5587-9637-45ea-9d12-97ba06c7011d","Type":"ContainerStarted","Data":"fca2a7791f245a50f1750f1851f30f177be70d6627c5db7e1fbd24d0f99898d4"} Nov 03 02:01:05.335775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:05.335665 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" event={"ID":"c8ad5587-9637-45ea-9d12-97ba06c7011d","Type":"ContainerStarted","Data":"851d5a54c471135a14121a7a29372e5d86b1a722832ce0a0c20ac1b7f9724733"} Nov 03 02:01:05.377222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:05.377176 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:05.384183 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:05.384125 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:05.451266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:05.451159 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" podStartSLOduration=1.451144724 podStartE2EDuration="1.451144724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:05.449840193 +0000 UTC m=+547.133949372" watchObservedRunningTime="2026-11-03 02:01:05.451144724 +0000 UTC m=+547.135253950" Nov 03 02:01:05.566777 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:05.566729 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" podStartSLOduration=2.566715662 podStartE2EDuration="2.566715662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:05.56582073 +0000 UTC m=+547.249929885" watchObservedRunningTime="2026-11-03 02:01:05.566715662 +0000 UTC m=+547.250824816" Nov 03 02:01:05.683628 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:01:05.680009 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c00682_647d_4422_8ba4_1149c73b3eee.slice/crio-823f53bc8557dd06e3bd0eadb75cbf1743e61dba2610588e744c3604ce7ce939 WatchSource:0}: Error finding container 823f53bc8557dd06e3bd0eadb75cbf1743e61dba2610588e744c3604ce7ce939: Status 404 returned error can't find the container with id 823f53bc8557dd06e3bd0eadb75cbf1743e61dba2610588e744c3604ce7ce939 Nov 03 02:01:05.683628 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:05.682890 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n"] Nov 03 02:01:05.685092 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:05.685049 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:05.685276 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:05.685246 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:05.685484 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:05.685458 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:06.082376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.082302 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:06.087113 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.087084 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:06.283529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.283495 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw"] Nov 03 02:01:06.287395 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.287375 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.341249 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.341172 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-080015ba-openshift-default-dockercfg-9mbmr\"" Nov 03 02:01:06.343059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.343034 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw"] Nov 03 02:01:06.344437 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.344405 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" event={"ID":"02c00682-647d-4422-8ba4-1149c73b3eee","Type":"ContainerStarted","Data":"389848060def630d22ec439f92e6be6ffe483fef0c732ccd5aafd2e854f82832"} Nov 03 02:01:06.344567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.344475 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" event={"ID":"02c00682-647d-4422-8ba4-1149c73b3eee","Type":"ContainerStarted","Data":"823f53bc8557dd06e3bd0eadb75cbf1743e61dba2610588e744c3604ce7ce939"} Nov 03 02:01:06.344712 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.344693 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:06.344801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.344745 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:06.345628 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.345610 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:06.345856 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.345836 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:06.404103 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.404064 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:06.426171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.422995 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4gr9\" (UniqueName: \"kubernetes.io/projected/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-kube-api-access-b4gr9\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.426171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.423051 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-credential-socket\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.426171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.423103 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-podinfo\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.426171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.423150 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-workload-certs\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.426171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.423185 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-envoy\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.426171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.423287 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-token\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.426171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.423322 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-workload-socket\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.426171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.423351 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istiod-ca-cert\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.426171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.423401 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-data\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.524601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.524570 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-workload-socket\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.524601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.524604 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istiod-ca-cert\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.524864 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.524659 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-data\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.525033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.525005 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-workload-socket\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.525222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.525024 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4gr9\" (UniqueName: \"kubernetes.io/projected/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-kube-api-access-b4gr9\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.525351 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.525067 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-data\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.525351 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.525304 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-credential-socket\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.525468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.525397 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-podinfo\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.525527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.525480 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-workload-certs\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.525585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.525531 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-envoy\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.525585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.525536 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-credential-socket\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.525701 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.525652 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istiod-ca-cert\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.525701 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.525661 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-token\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.526477 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.525990 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-workload-certs\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.528517 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.528492 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-podinfo\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.528623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.528608 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-envoy\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.559773 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.559737 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:06.582643 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.582620 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-token\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.582829 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.582806 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4gr9\" (UniqueName: \"kubernetes.io/projected/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-kube-api-access-b4gr9\") pod \"e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.600562 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.600508 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:06.720864 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.720835 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:06.760896 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:06.760844 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" podStartSLOduration=2.760829224 podStartE2EDuration="2.760829224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:06.757651777 +0000 UTC m=+548.441760934" watchObservedRunningTime="2026-11-03 02:01:06.760829224 +0000 UTC m=+548.444938378" Nov 03 02:01:07.037911 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:01:07.037880 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a70d009_4bbc_4a9e_8545_e47d9eb399b2.slice/crio-305fc85d02e79766c2118101fae4c13543f1f84799784f1d2680f916185a1ac7 WatchSource:0}: Error finding container 305fc85d02e79766c2118101fae4c13543f1f84799784f1d2680f916185a1ac7: Status 404 returned error can't find the container with id 305fc85d02e79766c2118101fae4c13543f1f84799784f1d2680f916185a1ac7 Nov 03 02:01:07.040315 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:07.040281 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:07.040424 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:07.040350 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:07.040424 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:07.040380 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:07.040489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:07.040439 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw"] Nov 03 02:01:07.244612 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:07.244584 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:07.249482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:07.249459 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:07.352279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:07.352179 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" event={"ID":"0a70d009-4bbc-4a9e-8545-e47d9eb399b2","Type":"ContainerStarted","Data":"060456b20ed83ac1d32dcd23512850dff35b64807b93966098322778e7dcc755"} Nov 03 02:01:07.352279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:07.352266 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" event={"ID":"0a70d009-4bbc-4a9e-8545-e47d9eb399b2","Type":"ContainerStarted","Data":"305fc85d02e79766c2118101fae4c13543f1f84799784f1d2680f916185a1ac7"} Nov 03 02:01:07.352744 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:07.352726 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:07.353466 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:07.353448 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:07.574177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:07.574146 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:07.599909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:07.599859 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" podStartSLOduration=1.5998452680000002 podStartE2EDuration="1.599845268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:07.598254308 +0000 UTC m=+549.282363454" watchObservedRunningTime="2026-11-03 02:01:07.599845268 +0000 UTC m=+549.283954482" Nov 03 02:01:07.600782 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:07.600670 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:07.774247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:07.774217 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:07.930734 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:07.930701 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:08.198516 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:08.198478 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:08.376669 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:08.376639 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:08.539299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:08.539209 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:08.605853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:08.605822 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:09.366651 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:09.366569 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:09.367588 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:09.367568 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:01:09.839617 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:09.839584 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:09.994316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:09.994282 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:10.193390 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.193362 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:10.649112 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.649083 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj"] Nov 03 02:01:10.653042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.653022 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.670321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.670297 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-credential-socket\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.670461 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.670348 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-podinfo\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.670461 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.670409 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-envoy\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.670461 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.670439 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-workload-socket\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.670613 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.670473 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-workload-certs\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.670613 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.670601 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-token\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.670701 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.670633 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-data\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.670701 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.670659 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istiod-ca-cert\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.670818 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.670800 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh824\" (UniqueName: \"kubernetes.io/projected/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-kube-api-access-wh824\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.708567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.708537 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-fbf24ed8-openshift-default-dockercfg-jqx6j\"" Nov 03 02:01:10.759243 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.759214 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj"] Nov 03 02:01:10.770331 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.770305 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:10.771408 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.771387 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh824\" (UniqueName: \"kubernetes.io/projected/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-kube-api-access-wh824\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.771503 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.771420 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-credential-socket\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.771503 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.771445 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-podinfo\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.771503 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.771477 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-envoy\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.771665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.771505 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-workload-socket\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.771665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.771543 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-workload-certs\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.771665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.771585 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-token\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.771665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.771606 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-data\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.771665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.771631 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istiod-ca-cert\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.771959 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.771938 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-credential-socket\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.772119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.772096 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-workload-socket\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.772394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.772350 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-data\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.772394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.772374 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istiod-ca-cert\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.772541 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.772437 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-workload-certs\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.774156 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.774135 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-podinfo\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.774266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.774165 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-envoy\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.828312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.828285 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-token\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.828447 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.828335 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh824\" (UniqueName: \"kubernetes.io/projected/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-kube-api-access-wh824\") pod \"e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:10.938361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.938285 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:10.966590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:10.966560 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:11.102968 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:11.102917 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:11.386990 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:01:11.386962 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c143c15_d9ea_4a97_81ed_04fa713fc5d7.slice/crio-79270060222d81bc5841112cd4a86c03fd1cd1544620c99ad2b69aa9ccc95e8c WatchSource:0}: Error finding container 79270060222d81bc5841112cd4a86c03fd1cd1544620c99ad2b69aa9ccc95e8c: Status 404 returned error can't find the container with id 79270060222d81bc5841112cd4a86c03fd1cd1544620c99ad2b69aa9ccc95e8c Nov 03 02:01:11.389541 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:11.389517 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj"] Nov 03 02:01:11.389759 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:11.389725 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:11.389813 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:11.389795 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:11.389857 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:11.389827 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:12.268320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:12.268294 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:12.384710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:12.384680 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" event={"ID":"9c143c15-d9ea-4a97-81ed-04fa713fc5d7","Type":"ContainerStarted","Data":"c9a8969a095d1893a0c94c5e669a090cf3dad6685712b9a6f020b328e6187d1a"} Nov 03 02:01:12.384851 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:12.384729 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" event={"ID":"9c143c15-d9ea-4a97-81ed-04fa713fc5d7","Type":"ContainerStarted","Data":"79270060222d81bc5841112cd4a86c03fd1cd1544620c99ad2b69aa9ccc95e8c"} Nov 03 02:01:12.455491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:12.455461 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:12.524286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:12.516345 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" podStartSLOduration=2.516321574 podStartE2EDuration="2.516321574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:12.515632911 +0000 UTC m=+554.199742086" watchObservedRunningTime="2026-11-03 02:01:12.516321574 +0000 UTC m=+554.200430729" Nov 03 02:01:12.602180 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:12.602140 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:12.967430 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:12.967402 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:12.972305 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:12.972284 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:13.005974 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:01:13.005940 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:01:13.390131 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:13.390033 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:13.391063 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:13.391042 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:13.603716 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:13.603682 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:13.796227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:13.796172 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:14.086292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:14.086175 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:14.244483 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:14.244452 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:14.487980 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:14.487945 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:15.573184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:15.573149 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:15.741222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:15.741175 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:17.734963 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:17.734930 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:17.961326 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:17.961296 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:18.130451 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:18.127728 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:18.282620 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:18.282587 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:19.242617 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:19.242580 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:19.404586 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:19.404551 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:19.556980 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:19.556875 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:20.542560 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:20.542527 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:20.720456 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:20.720421 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:21.089930 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:21.089886 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh"] Nov 03 02:01:21.093875 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:21.093851 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh" Nov 03 02:01:21.146643 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:21.146610 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh"] Nov 03 02:01:21.193348 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:21.193306 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb6l4\" (UniqueName: \"kubernetes.io/projected/2670a524-52ac-4c55-9936-5873da1aa35d-kube-api-access-tb6l4\") pod \"payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh\" (UID: \"2670a524-52ac-4c55-9936-5873da1aa35d\") " pod="openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh" Nov 03 02:01:21.193504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:21.193402 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2670a524-52ac-4c55-9936-5873da1aa35d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh\" (UID: \"2670a524-52ac-4c55-9936-5873da1aa35d\") " pod="openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh" Nov 03 02:01:21.194019 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:21.193985 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-7683f606\"" Nov 03 02:01:21.194019 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:21.193994 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-7683f606-dockercfg-vxjhq\"" Nov 03 02:01:21.294787 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:21.294744 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2670a524-52ac-4c55-9936-5873da1aa35d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh\" (UID: \"2670a524-52ac-4c55-9936-5873da1aa35d\") " pod="openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh" Nov 03 02:01:21.294953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:21.294910 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb6l4\" (UniqueName: \"kubernetes.io/projected/2670a524-52ac-4c55-9936-5873da1aa35d-kube-api-access-tb6l4\") pod \"payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh\" (UID: \"2670a524-52ac-4c55-9936-5873da1aa35d\") " pod="openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh" Nov 03 02:01:21.295559 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:21.295541 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2670a524-52ac-4c55-9936-5873da1aa35d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh\" (UID: \"2670a524-52ac-4c55-9936-5873da1aa35d\") " pod="openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh" Nov 03 02:01:21.351861 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:21.351784 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb6l4\" (UniqueName: \"kubernetes.io/projected/2670a524-52ac-4c55-9936-5873da1aa35d-kube-api-access-tb6l4\") pod \"payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh\" (UID: \"2670a524-52ac-4c55-9936-5873da1aa35d\") " pod="openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh" Nov 03 02:01:21.407807 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:21.407772 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh" Nov 03 02:01:21.773068 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:01:21.773035 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2670a524_52ac_4c55_9936_5873da1aa35d.slice/crio-a1103386689a52523c04d15395874048487eee66650dbcdf08553b334e8e6d56 WatchSource:0}: Error finding container a1103386689a52523c04d15395874048487eee66650dbcdf08553b334e8e6d56: Status 404 returned error can't find the container with id a1103386689a52523c04d15395874048487eee66650dbcdf08553b334e8e6d56 Nov 03 02:01:21.776653 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:21.776625 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh"] Nov 03 02:01:22.456145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:22.456104 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh" event={"ID":"2670a524-52ac-4c55-9936-5873da1aa35d","Type":"ContainerStarted","Data":"c28d38ab94075ce0d468f71a917446f66ae879aa82a5c836040a2426a9229fc9"} Nov 03 02:01:22.456361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:22.456164 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh" event={"ID":"2670a524-52ac-4c55-9936-5873da1aa35d","Type":"ContainerStarted","Data":"a1103386689a52523c04d15395874048487eee66650dbcdf08553b334e8e6d56"} Nov 03 02:01:22.456361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:22.456187 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh" Nov 03 02:01:22.574365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:22.574311 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh" podStartSLOduration=1.574297727 podStartE2EDuration="1.574297727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:22.573161375 +0000 UTC m=+564.257270531" watchObservedRunningTime="2026-11-03 02:01:22.574297727 +0000 UTC m=+564.258406881" Nov 03 02:01:23.890775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:23.890739 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:24.049722 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:24.049686 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:24.192781 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:24.192753 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:28.859417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:28.857898 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:30.198109 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.198044 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:30.384869 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.384837 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:30.597978 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.597902 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m"] Nov 03 02:01:30.601887 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.601871 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m" Nov 03 02:01:30.655241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.655206 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m"] Nov 03 02:01:30.702096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.702066 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-fbf24ed8\"" Nov 03 02:01:30.702328 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.702287 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-fbf24ed8-dockercfg-58n42\"" Nov 03 02:01:30.724346 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.724315 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a0798f-c015-4dc1-b468-c764722cb2ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m\" (UID: \"b0a0798f-c015-4dc1-b468-c764722cb2ff\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m" Nov 03 02:01:30.724524 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.724454 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdm8h\" (UniqueName: \"kubernetes.io/projected/b0a0798f-c015-4dc1-b468-c764722cb2ff-kube-api-access-mdm8h\") pod \"payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m\" (UID: \"b0a0798f-c015-4dc1-b468-c764722cb2ff\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m" Nov 03 02:01:30.754867 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.754840 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h"] Nov 03 02:01:30.758903 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.758884 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h" Nov 03 02:01:30.796511 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.796483 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h"] Nov 03 02:01:30.825288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.825172 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86377347-5a8a-4be2-ba1d-a333b78c57ec-plugins-config-volume\") pod \"payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h\" (UID: \"86377347-5a8a-4be2-ba1d-a333b78c57ec\") " pod="openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h" Nov 03 02:01:30.825433 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.825342 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt94t\" (UniqueName: \"kubernetes.io/projected/86377347-5a8a-4be2-ba1d-a333b78c57ec-kube-api-access-wt94t\") pod \"payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h\" (UID: \"86377347-5a8a-4be2-ba1d-a333b78c57ec\") " pod="openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h" Nov 03 02:01:30.825497 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.825443 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a0798f-c015-4dc1-b468-c764722cb2ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m\" (UID: \"b0a0798f-c015-4dc1-b468-c764722cb2ff\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m" Nov 03 02:01:30.826347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.826320 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a0798f-c015-4dc1-b468-c764722cb2ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m\" (UID: \"b0a0798f-c015-4dc1-b468-c764722cb2ff\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m" Nov 03 02:01:30.826504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.826486 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdm8h\" (UniqueName: \"kubernetes.io/projected/b0a0798f-c015-4dc1-b468-c764722cb2ff-kube-api-access-mdm8h\") pod \"payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m\" (UID: \"b0a0798f-c015-4dc1-b468-c764722cb2ff\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m" Nov 03 02:01:30.883899 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.883821 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdm8h\" (UniqueName: \"kubernetes.io/projected/b0a0798f-c015-4dc1-b468-c764722cb2ff-kube-api-access-mdm8h\") pod \"payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m\" (UID: \"b0a0798f-c015-4dc1-b468-c764722cb2ff\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m" Nov 03 02:01:30.915658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.915630 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m" Nov 03 02:01:30.927593 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.927558 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86377347-5a8a-4be2-ba1d-a333b78c57ec-plugins-config-volume\") pod \"payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h\" (UID: \"86377347-5a8a-4be2-ba1d-a333b78c57ec\") " pod="openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h" Nov 03 02:01:30.927748 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.927626 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt94t\" (UniqueName: \"kubernetes.io/projected/86377347-5a8a-4be2-ba1d-a333b78c57ec-kube-api-access-wt94t\") pod \"payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h\" (UID: \"86377347-5a8a-4be2-ba1d-a333b78c57ec\") " pod="openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h" Nov 03 02:01:30.928860 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.928813 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86377347-5a8a-4be2-ba1d-a333b78c57ec-plugins-config-volume\") pod \"payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h\" (UID: \"86377347-5a8a-4be2-ba1d-a333b78c57ec\") " pod="openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h" Nov 03 02:01:30.991127 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:30.991098 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt94t\" (UniqueName: \"kubernetes.io/projected/86377347-5a8a-4be2-ba1d-a333b78c57ec-kube-api-access-wt94t\") pod \"payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h\" (UID: \"86377347-5a8a-4be2-ba1d-a333b78c57ec\") " pod="openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h" Nov 03 02:01:31.072814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:31.072521 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h" Nov 03 02:01:31.302164 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:01:31.302130 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a0798f_c015_4dc1_b468_c764722cb2ff.slice/crio-7d1be1587326b094e7f2fc5c7d8cb3c61243f6e7c035919aec13804fce474c72 WatchSource:0}: Error finding container 7d1be1587326b094e7f2fc5c7d8cb3c61243f6e7c035919aec13804fce474c72: Status 404 returned error can't find the container with id 7d1be1587326b094e7f2fc5c7d8cb3c61243f6e7c035919aec13804fce474c72 Nov 03 02:01:31.305330 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:31.305301 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m"] Nov 03 02:01:31.459426 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:01:31.459389 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86377347_5a8a_4be2_ba1d_a333b78c57ec.slice/crio-d8b851624f1cc63a2a2ddfbe65685aa28b10de7677b767aa913fd9bb55be1e3a WatchSource:0}: Error finding container d8b851624f1cc63a2a2ddfbe65685aa28b10de7677b767aa913fd9bb55be1e3a: Status 404 returned error can't find the container with id d8b851624f1cc63a2a2ddfbe65685aa28b10de7677b767aa913fd9bb55be1e3a Nov 03 02:01:31.459905 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:31.459882 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h"] Nov 03 02:01:31.553374 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:31.553335 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h" event={"ID":"86377347-5a8a-4be2-ba1d-a333b78c57ec","Type":"ContainerStarted","Data":"3d325b4a4f9bae73e7768bd0e3d10a22fc244bccd145ae472d59873e89e1dd9d"} Nov 03 02:01:31.553573 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:31.553402 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h" event={"ID":"86377347-5a8a-4be2-ba1d-a333b78c57ec","Type":"ContainerStarted","Data":"d8b851624f1cc63a2a2ddfbe65685aa28b10de7677b767aa913fd9bb55be1e3a"} Nov 03 02:01:31.554298 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:31.554274 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h" Nov 03 02:01:31.556324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:31.556297 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m" event={"ID":"b0a0798f-c015-4dc1-b468-c764722cb2ff","Type":"ContainerStarted","Data":"57158d80ac6b7042e10b45daba76d600dbb0aaac38f07ecd1c21075eda39d0ff"} Nov 03 02:01:31.556434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:31.556348 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m" event={"ID":"b0a0798f-c015-4dc1-b468-c764722cb2ff","Type":"ContainerStarted","Data":"7d1be1587326b094e7f2fc5c7d8cb3c61243f6e7c035919aec13804fce474c72"} Nov 03 02:01:31.556931 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:31.556912 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m" Nov 03 02:01:31.670040 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:31.669969 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h" podStartSLOduration=1.669949829 podStartE2EDuration="1.669949829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:31.667268368 +0000 UTC m=+573.351377525" watchObservedRunningTime="2026-11-03 02:01:31.669949829 +0000 UTC m=+573.354058984" Nov 03 02:01:31.782278 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:31.782151 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m" podStartSLOduration=1.78212007 podStartE2EDuration="1.78212007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:31.781971428 +0000 UTC m=+573.466080584" watchObservedRunningTime="2026-11-03 02:01:31.78212007 +0000 UTC m=+573.466229252" Nov 03 02:01:33.465713 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:33.465683 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh" Nov 03 02:01:33.580376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:33.580337 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:33.728325 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:33.728245 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:33.894975 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:33.894933 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:34.578941 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:34.578908 5463 generic.go:356] "Generic (PLEG): container finished" podID="c8ad5587-9637-45ea-9d12-97ba06c7011d" containerID="fca2a7791f245a50f1750f1851f30f177be70d6627c5db7e1fbd24d0f99898d4" exitCode=0 Nov 03 02:01:34.579499 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:34.578984 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" event={"ID":"c8ad5587-9637-45ea-9d12-97ba06c7011d","Type":"ContainerDied","Data":"fca2a7791f245a50f1750f1851f30f177be70d6627c5db7e1fbd24d0f99898d4"} Nov 03 02:01:34.579499 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:34.579457 5463 scope.go:122] "RemoveContainer" containerID="fca2a7791f245a50f1750f1851f30f177be70d6627c5db7e1fbd24d0f99898d4" Nov 03 02:01:34.580841 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:34.580807 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:34.580960 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:34.580894 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:34.580960 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:34.580945 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:34.581074 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:34.581059 5463 generic.go:356] "Generic (PLEG): container finished" podID="02c00682-647d-4422-8ba4-1149c73b3eee" containerID="389848060def630d22ec439f92e6be6ffe483fef0c732ccd5aafd2e854f82832" exitCode=0 Nov 03 02:01:34.581129 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:34.581108 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" event={"ID":"02c00682-647d-4422-8ba4-1149c73b3eee","Type":"ContainerDied","Data":"389848060def630d22ec439f92e6be6ffe483fef0c732ccd5aafd2e854f82832"} Nov 03 02:01:34.581480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:34.581428 5463 scope.go:122] "RemoveContainer" containerID="389848060def630d22ec439f92e6be6ffe483fef0c732ccd5aafd2e854f82832" Nov 03 02:01:34.582491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:34.582459 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:34.582588 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:34.582536 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:34.582650 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:34.582585 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:35.624217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:35.618462 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" event={"ID":"c8ad5587-9637-45ea-9d12-97ba06c7011d","Type":"ContainerStarted","Data":"c53f1bc2b17d01fc1368b544d60566b7a369ff799a32677107ce33b6a5fb7f0d"} Nov 03 02:01:35.626659 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:35.626635 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" event={"ID":"02c00682-647d-4422-8ba4-1149c73b3eee","Type":"ContainerStarted","Data":"ffc6952367e996ae428f07f3f3c81f6f73c0ef5f4336eb62e471b13e4b928d02"} Nov 03 02:01:36.084216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:36.083347 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:36.092235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:36.091338 5463 patch_prober.go:29] interesting pod/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 02:01:36.092235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:36.091412 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" podUID="c8ad5587-9637-45ea-9d12-97ba06c7011d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 02:01:36.244111 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:36.244074 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:36.244912 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:36.244886 5463 patch_prober.go:29] interesting pod/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 03 02:01:36.244978 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:36.244945 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" podUID="02c00682-647d-4422-8ba4-1149c73b3eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 03 02:01:36.345256 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:36.345141 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:37.086214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:37.085626 5463 patch_prober.go:29] interesting pod/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 02:01:37.086214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:37.085688 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" podUID="c8ad5587-9637-45ea-9d12-97ba06c7011d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 02:01:37.244209 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:37.244148 5463 patch_prober.go:29] interesting pod/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 03 02:01:37.244387 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:37.244249 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" podUID="02c00682-647d-4422-8ba4-1149c73b3eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 03 02:01:37.353648 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:37.353552 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:38.085338 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:38.085299 5463 patch_prober.go:29] interesting pod/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 02:01:38.085627 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:38.085593 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" podUID="c8ad5587-9637-45ea-9d12-97ba06c7011d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 02:01:38.245030 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:38.245001 5463 patch_prober.go:29] interesting pod/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 03 02:01:38.245510 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:38.245480 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" podUID="02c00682-647d-4422-8ba4-1149c73b3eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 03 02:01:38.529225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:38.528738 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:39.089247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:39.089213 5463 patch_prober.go:29] interesting pod/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 02:01:39.089532 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:39.089502 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" podUID="c8ad5587-9637-45ea-9d12-97ba06c7011d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 02:01:39.244176 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:39.244148 5463 patch_prober.go:29] interesting pod/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 03 02:01:39.244407 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:39.244386 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" podUID="02c00682-647d-4422-8ba4-1149c73b3eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 03 02:01:40.085622 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:40.085580 5463 patch_prober.go:29] interesting pod/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 02:01:40.086272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:40.086227 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" podUID="c8ad5587-9637-45ea-9d12-97ba06c7011d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 02:01:40.244271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:40.244241 5463 patch_prober.go:29] interesting pod/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 03 02:01:40.244483 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:40.244463 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" podUID="02c00682-647d-4422-8ba4-1149c73b3eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 03 02:01:41.087801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:41.087766 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:41.088963 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:41.088937 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:01:41.248942 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:41.248911 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:41.250254 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:41.250224 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:01:43.009428 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:01:43.009382 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:01:43.202501 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:43.202454 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:43.368305 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:43.368213 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:43.516679 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:43.516651 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:43.577322 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:43.577275 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m" Nov 03 02:01:43.578493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:43.578468 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h" Nov 03 02:01:44.737113 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:44.737080 5463 generic.go:356] "Generic (PLEG): container finished" podID="cc06054f-00b6-4308-ac4b-a315a36ef0ba" containerID="34a8539a377ca9d6fd6acaa63603a5715158a5be41a13052018289cd55c66fe3" exitCode=0 Nov 03 02:01:44.737559 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:44.737155 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" event={"ID":"cc06054f-00b6-4308-ac4b-a315a36ef0ba","Type":"ContainerDied","Data":"34a8539a377ca9d6fd6acaa63603a5715158a5be41a13052018289cd55c66fe3"} Nov 03 02:01:44.737600 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:44.737564 5463 scope.go:122] "RemoveContainer" containerID="34a8539a377ca9d6fd6acaa63603a5715158a5be41a13052018289cd55c66fe3" Nov 03 02:01:44.738891 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:44.738860 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:44.739019 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:44.738942 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:44.739019 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:44.738991 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:45.744997 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:45.744959 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" event={"ID":"cc06054f-00b6-4308-ac4b-a315a36ef0ba","Type":"ContainerStarted","Data":"81a2907b67c30e18ee93bb06f607d9787c4601f915562bf6cc77791958edf854"} Nov 03 02:01:46.387595 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:46.387562 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:46.387944 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:46.387921 5463 patch_prober.go:29] interesting pod/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 02:01:46.388112 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:46.388085 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" podUID="cc06054f-00b6-4308-ac4b-a315a36ef0ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 02:01:47.382215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:47.381648 5463 patch_prober.go:29] interesting pod/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 02:01:47.382215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:47.381711 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" podUID="cc06054f-00b6-4308-ac4b-a315a36ef0ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 02:01:48.378596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:48.378559 5463 patch_prober.go:29] interesting pod/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 02:01:48.378877 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:48.378848 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" podUID="cc06054f-00b6-4308-ac4b-a315a36ef0ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 02:01:48.645812 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:48.645784 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:49.376960 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:49.376866 5463 patch_prober.go:29] interesting pod/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 02:01:49.376960 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:49.376923 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" podUID="cc06054f-00b6-4308-ac4b-a315a36ef0ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 02:01:50.376709 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:50.376671 5463 patch_prober.go:29] interesting pod/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 02:01:50.377176 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:50.376724 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" podUID="cc06054f-00b6-4308-ac4b-a315a36ef0ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 02:01:51.345292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:51.345260 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:51.381568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:51.381535 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:51.817096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:51.817065 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:01:52.823532 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:52.823452 5463 generic.go:356] "Generic (PLEG): container finished" podID="9c143c15-d9ea-4a97-81ed-04fa713fc5d7" containerID="c9a8969a095d1893a0c94c5e669a090cf3dad6685712b9a6f020b328e6187d1a" exitCode=0 Nov 03 02:01:52.823923 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:52.823529 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" event={"ID":"9c143c15-d9ea-4a97-81ed-04fa713fc5d7","Type":"ContainerDied","Data":"c9a8969a095d1893a0c94c5e669a090cf3dad6685712b9a6f020b328e6187d1a"} Nov 03 02:01:52.824153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:52.824131 5463 scope.go:122] "RemoveContainer" containerID="c9a8969a095d1893a0c94c5e669a090cf3dad6685712b9a6f020b328e6187d1a" Nov 03 02:01:52.825318 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:52.825290 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:52.825384 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:52.825359 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:52.825422 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:52.825393 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:01:53.366213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:53.357640 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:53.831064 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:53.831028 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" event={"ID":"9c143c15-d9ea-4a97-81ed-04fa713fc5d7","Type":"ContainerStarted","Data":"21f72345861d4eb204999c98f479b7fd0eaa645e91a31b9e219507253d32cd9e"} Nov 03 02:01:53.971109 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:53.971079 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:53.971405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:53.971320 5463 patch_prober.go:29] interesting pod/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 02:01:53.971546 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:53.971525 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" podUID="9c143c15-d9ea-4a97-81ed-04fa713fc5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 02:01:54.727964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:54.727921 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89"] Nov 03 02:01:54.731228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:54.731209 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89" Nov 03 02:01:54.801220 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:54.786092 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89"] Nov 03 02:01:54.801403 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:54.798550 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:01:54.801403 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:54.799239 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:01:54.801403 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:54.799287 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:01:54.801403 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:54.799354 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:01:54.834501 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:54.834476 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e391962-85cb-46bd-a13e-d53976cb0b8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7683f606-75x89\" (UID: \"0e391962-85cb-46bd-a13e-d53976cb0b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89" Nov 03 02:01:54.834974 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:54.834554 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2r7f\" (UniqueName: \"kubernetes.io/projected/0e391962-85cb-46bd-a13e-d53976cb0b8b-kube-api-access-v2r7f\") pod \"maas-api-revoke-keys-e2e-migrate-7683f606-75x89\" (UID: \"0e391962-85cb-46bd-a13e-d53976cb0b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89" Nov 03 02:01:54.834974 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:54.834724 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e391962-85cb-46bd-a13e-d53976cb0b8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7683f606-75x89\" (UID: \"0e391962-85cb-46bd-a13e-d53976cb0b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89" Nov 03 02:01:54.935530 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:54.935490 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e391962-85cb-46bd-a13e-d53976cb0b8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7683f606-75x89\" (UID: \"0e391962-85cb-46bd-a13e-d53976cb0b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89" Nov 03 02:01:54.935530 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:54.935529 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e391962-85cb-46bd-a13e-d53976cb0b8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7683f606-75x89\" (UID: \"0e391962-85cb-46bd-a13e-d53976cb0b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89" Nov 03 02:01:54.935792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:54.935570 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2r7f\" (UniqueName: \"kubernetes.io/projected/0e391962-85cb-46bd-a13e-d53976cb0b8b-kube-api-access-v2r7f\") pod \"maas-api-revoke-keys-e2e-migrate-7683f606-75x89\" (UID: \"0e391962-85cb-46bd-a13e-d53976cb0b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89" Nov 03 02:01:54.936379 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:54.936351 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e391962-85cb-46bd-a13e-d53976cb0b8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7683f606-75x89\" (UID: \"0e391962-85cb-46bd-a13e-d53976cb0b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89" Nov 03 02:01:54.938617 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:54.938595 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e391962-85cb-46bd-a13e-d53976cb0b8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7683f606-75x89\" (UID: \"0e391962-85cb-46bd-a13e-d53976cb0b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89" Nov 03 02:01:54.971224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:54.969953 5463 patch_prober.go:29] interesting pod/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 02:01:54.971224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:54.970022 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" podUID="9c143c15-d9ea-4a97-81ed-04fa713fc5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 02:01:55.005359 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:55.005284 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2r7f\" (UniqueName: \"kubernetes.io/projected/0e391962-85cb-46bd-a13e-d53976cb0b8b-kube-api-access-v2r7f\") pod \"maas-api-revoke-keys-e2e-migrate-7683f606-75x89\" (UID: \"0e391962-85cb-46bd-a13e-d53976cb0b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89" Nov 03 02:01:55.043677 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:55.043647 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89" Nov 03 02:01:55.499452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:55.499326 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89"] Nov 03 02:01:55.843643 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:55.843565 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89" event={"ID":"0e391962-85cb-46bd-a13e-d53976cb0b8b","Type":"ContainerStarted","Data":"2128a3c8ba4050ebad94dacd2813820dd7bf9bc1aedb1639a6affe666b6692e6"} Nov 03 02:01:55.973215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:55.968650 5463 patch_prober.go:29] interesting pod/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 02:01:55.973215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:55.968704 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" podUID="9c143c15-d9ea-4a97-81ed-04fa713fc5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 02:01:56.876220 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:56.876106 5463 generic.go:356] "Generic (PLEG): container finished" podID="0e391962-85cb-46bd-a13e-d53976cb0b8b" containerID="a3d3f9707c98d7bcb9c1c4961535d71422c391bc1728ea80bba9a6466569dc55" exitCode=0 Nov 03 02:01:56.876913 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:56.876890 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89" event={"ID":"0e391962-85cb-46bd-a13e-d53976cb0b8b","Type":"ContainerDied","Data":"a3d3f9707c98d7bcb9c1c4961535d71422c391bc1728ea80bba9a6466569dc55"} Nov 03 02:01:56.980364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:56.980328 5463 patch_prober.go:29] interesting pod/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 02:01:56.980634 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:56.980607 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" podUID="9c143c15-d9ea-4a97-81ed-04fa713fc5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 02:01:57.970654 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:57.970614 5463 patch_prober.go:29] interesting pod/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 02:01:57.972708 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:57.972674 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" podUID="9c143c15-d9ea-4a97-81ed-04fa713fc5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 02:01:58.173434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:58.173366 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89" Nov 03 02:01:58.322305 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:58.322267 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e391962-85cb-46bd-a13e-d53976cb0b8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e391962-85cb-46bd-a13e-d53976cb0b8b-maas-api-xks-ca\") pod \"0e391962-85cb-46bd-a13e-d53976cb0b8b\" (UID: \"0e391962-85cb-46bd-a13e-d53976cb0b8b\") " Nov 03 02:01:58.322466 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:58.322407 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e391962-85cb-46bd-a13e-d53976cb0b8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e391962-85cb-46bd-a13e-d53976cb0b8b-maas-api-openshift-ca\") pod \"0e391962-85cb-46bd-a13e-d53976cb0b8b\" (UID: \"0e391962-85cb-46bd-a13e-d53976cb0b8b\") " Nov 03 02:01:58.322466 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:58.322432 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e391962-85cb-46bd-a13e-d53976cb0b8b-kube-api-access-v2r7f\" (UniqueName: \"kubernetes.io/projected/0e391962-85cb-46bd-a13e-d53976cb0b8b-kube-api-access-v2r7f\") pod \"0e391962-85cb-46bd-a13e-d53976cb0b8b\" (UID: \"0e391962-85cb-46bd-a13e-d53976cb0b8b\") " Nov 03 02:01:58.323038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:58.323008 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e391962-85cb-46bd-a13e-d53976cb0b8b-maas-api-openshift-ca" pod "0e391962-85cb-46bd-a13e-d53976cb0b8b" (UID: "0e391962-85cb-46bd-a13e-d53976cb0b8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:01:58.324730 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:58.324703 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e391962-85cb-46bd-a13e-d53976cb0b8b-maas-api-xks-ca" pod "0e391962-85cb-46bd-a13e-d53976cb0b8b" (UID: "0e391962-85cb-46bd-a13e-d53976cb0b8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:01:58.324825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:58.324784 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e391962-85cb-46bd-a13e-d53976cb0b8b-kube-api-access-v2r7f" pod "0e391962-85cb-46bd-a13e-d53976cb0b8b" (UID: "0e391962-85cb-46bd-a13e-d53976cb0b8b"). InnerVolumeSpecName "kube-api-access-v2r7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:01:58.390410 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:58.390375 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:58.423553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:58.423466 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e391962-85cb-46bd-a13e-d53976cb0b8b-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:58.423553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:58.423496 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2r7f\" (UniqueName: \"kubernetes.io/projected/0e391962-85cb-46bd-a13e-d53976cb0b8b-kube-api-access-v2r7f\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:58.423553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:58.423514 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e391962-85cb-46bd-a13e-d53976cb0b8b-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:58.476425 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:58.476392 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:58.531401 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:58.531374 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:01:58.532497 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:58.532478 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:01:58.912875 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:58.912850 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89" Nov 03 02:01:58.913086 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:58.912876 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89" event={"ID":"0e391962-85cb-46bd-a13e-d53976cb0b8b","Type":"ContainerDied","Data":"2128a3c8ba4050ebad94dacd2813820dd7bf9bc1aedb1639a6affe666b6692e6"} Nov 03 02:01:58.913086 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:58.912938 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2128a3c8ba4050ebad94dacd2813820dd7bf9bc1aedb1639a6affe666b6692e6" Nov 03 02:01:58.967587 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:58.967552 5463 patch_prober.go:29] interesting pod/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 02:01:58.967747 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:58.967624 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" podUID="9c143c15-d9ea-4a97-81ed-04fa713fc5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 02:01:59.972098 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:59.972066 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:01:59.973139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:01:59.973116 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:02:00.644150 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:00.641381 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:00.838301 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:00.838272 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:01.935070 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:01.935037 5463 generic.go:356] "Generic (PLEG): container finished" podID="0a70d009-4bbc-4a9e-8545-e47d9eb399b2" containerID="060456b20ed83ac1d32dcd23512850dff35b64807b93966098322778e7dcc755" exitCode=0 Nov 03 02:02:01.935533 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:01.935107 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" event={"ID":"0a70d009-4bbc-4a9e-8545-e47d9eb399b2","Type":"ContainerDied","Data":"060456b20ed83ac1d32dcd23512850dff35b64807b93966098322778e7dcc755"} Nov 03 02:02:01.935533 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:01.935510 5463 scope.go:122] "RemoveContainer" containerID="060456b20ed83ac1d32dcd23512850dff35b64807b93966098322778e7dcc755" Nov 03 02:02:01.936830 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:01.936796 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:01.936954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:01.936884 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:01.936954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:01.936940 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:02.943948 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:02.943914 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" event={"ID":"0a70d009-4bbc-4a9e-8545-e47d9eb399b2","Type":"ContainerStarted","Data":"64d85d1d0cb8ef879c275fe5766e0110602db66dfb897f74c375806d7c390270"} Nov 03 02:02:03.601296 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:03.601259 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:02:03.601490 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:03.601470 5463 patch_prober.go:29] interesting pod/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 03 02:02:03.601541 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:03.601517 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" podUID="0a70d009-4bbc-4a9e-8545-e47d9eb399b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 03 02:02:03.619500 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:03.619471 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:04.601491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:04.601457 5463 patch_prober.go:29] interesting pod/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 03 02:02:04.601904 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:04.601515 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" podUID="0a70d009-4bbc-4a9e-8545-e47d9eb399b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 03 02:02:05.601440 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:05.601409 5463 patch_prober.go:29] interesting pod/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 03 02:02:05.601607 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:05.601455 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" podUID="0a70d009-4bbc-4a9e-8545-e47d9eb399b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 03 02:02:06.143364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:06.143329 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:06.601396 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:06.601312 5463 patch_prober.go:29] interesting pod/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 03 02:02:06.601396 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:06.601382 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" podUID="0a70d009-4bbc-4a9e-8545-e47d9eb399b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 03 02:02:06.727814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:06.727780 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:07.523624 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:07.523590 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh"] Nov 03 02:02:07.601416 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:07.601384 5463 patch_prober.go:29] interesting pod/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 03 02:02:07.601579 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:07.601446 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" podUID="0a70d009-4bbc-4a9e-8545-e47d9eb399b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 03 02:02:07.907390 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:07.907353 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:08.095227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:08.095149 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:08.265364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:08.265280 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:08.605928 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:08.605848 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:02:08.606241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:08.606221 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:02:08.606849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:08.606828 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:02:09.694410 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:09.694367 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:09.924167 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:09.924130 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:10.074843 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:10.074763 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:10.327251 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:10.327143 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt"] Nov 03 02:02:10.327426 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:10.327394 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" podUID="c8ad5587-9637-45ea-9d12-97ba06c7011d" containerName="istio-proxy" containerID="cri-o://c53f1bc2b17d01fc1368b544d60566b7a369ff799a32677107ce33b6a5fb7f0d" gracePeriod=30 Nov 03 02:02:10.337239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:10.337180 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:11.090810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:11.090775 5463 patch_prober.go:29] interesting pod/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:02:11.091217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:11.090846 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" podUID="c8ad5587-9637-45ea-9d12-97ba06c7011d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:02:11.184299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:11.184262 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:11.799257 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:11.799222 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:12.022150 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:12.022119 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:12.524457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:12.524419 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh" podUID="2670a524-52ac-4c55-9936-5873da1aa35d" containerName="payload-processing" containerID="cri-o://c28d38ab94075ce0d468f71a917446f66ae879aa82a5c836040a2426a9229fc9" gracePeriod=25 Nov 03 02:02:12.780841 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:12.780785 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh" Nov 03 02:02:12.784470 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:12.784450 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2670a524-52ac-4c55-9936-5873da1aa35d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2670a524-52ac-4c55-9936-5873da1aa35d-plugins-config-volume\") pod \"2670a524-52ac-4c55-9936-5873da1aa35d\" (UID: \"2670a524-52ac-4c55-9936-5873da1aa35d\") " Nov 03 02:02:12.784579 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:12.784546 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2670a524-52ac-4c55-9936-5873da1aa35d-kube-api-access-tb6l4\" (UniqueName: \"kubernetes.io/projected/2670a524-52ac-4c55-9936-5873da1aa35d-kube-api-access-tb6l4\") pod \"2670a524-52ac-4c55-9936-5873da1aa35d\" (UID: \"2670a524-52ac-4c55-9936-5873da1aa35d\") " Nov 03 02:02:12.793854 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:12.785076 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2670a524-52ac-4c55-9936-5873da1aa35d-plugins-config-volume" pod "2670a524-52ac-4c55-9936-5873da1aa35d" (UID: "2670a524-52ac-4c55-9936-5873da1aa35d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:02:12.793854 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:12.791812 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2670a524-52ac-4c55-9936-5873da1aa35d-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:12.794029 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:12.793946 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2670a524-52ac-4c55-9936-5873da1aa35d-kube-api-access-tb6l4" pod "2670a524-52ac-4c55-9936-5873da1aa35d" (UID: "2670a524-52ac-4c55-9936-5873da1aa35d"). InnerVolumeSpecName "kube-api-access-tb6l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:12.893455 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:12.893425 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb6l4\" (UniqueName: \"kubernetes.io/projected/2670a524-52ac-4c55-9936-5873da1aa35d-kube-api-access-tb6l4\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:13.008384 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:02:13.008349 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:02:13.075915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:13.075835 5463 generic.go:356] "Generic (PLEG): container finished" podID="2670a524-52ac-4c55-9936-5873da1aa35d" containerID="c28d38ab94075ce0d468f71a917446f66ae879aa82a5c836040a2426a9229fc9" exitCode=0 Nov 03 02:02:13.075915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:13.075887 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh" Nov 03 02:02:13.075915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:13.075893 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh" event={"ID":"2670a524-52ac-4c55-9936-5873da1aa35d","Type":"ContainerDied","Data":"c28d38ab94075ce0d468f71a917446f66ae879aa82a5c836040a2426a9229fc9"} Nov 03 02:02:13.076128 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:13.075964 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh" event={"ID":"2670a524-52ac-4c55-9936-5873da1aa35d","Type":"ContainerDied","Data":"a1103386689a52523c04d15395874048487eee66650dbcdf08553b334e8e6d56"} Nov 03 02:02:13.076128 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:13.075988 5463 scope.go:122] "RemoveContainer" containerID="c28d38ab94075ce0d468f71a917446f66ae879aa82a5c836040a2426a9229fc9" Nov 03 02:02:13.086055 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:13.086036 5463 scope.go:122] "RemoveContainer" containerID="c28d38ab94075ce0d468f71a917446f66ae879aa82a5c836040a2426a9229fc9" Nov 03 02:02:13.086373 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:02:13.086350 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c28d38ab94075ce0d468f71a917446f66ae879aa82a5c836040a2426a9229fc9\": container with ID starting with c28d38ab94075ce0d468f71a917446f66ae879aa82a5c836040a2426a9229fc9 not found: ID does not exist" containerID="c28d38ab94075ce0d468f71a917446f66ae879aa82a5c836040a2426a9229fc9" Nov 03 02:02:13.086464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:13.086384 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c28d38ab94075ce0d468f71a917446f66ae879aa82a5c836040a2426a9229fc9"} err="failed to get container status \"c28d38ab94075ce0d468f71a917446f66ae879aa82a5c836040a2426a9229fc9\": rpc error: code = NotFound desc = could not find container \"c28d38ab94075ce0d468f71a917446f66ae879aa82a5c836040a2426a9229fc9\": container with ID starting with c28d38ab94075ce0d468f71a917446f66ae879aa82a5c836040a2426a9229fc9 not found: ID does not exist" Nov 03 02:02:13.239727 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:13.239692 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh"] Nov 03 02:02:13.243168 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:13.243145 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7683f606-796c579bbf-xf7dh"] Nov 03 02:02:14.325858 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:14.325815 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:14.443282 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:14.443253 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2670a524-52ac-4c55-9936-5873da1aa35d" path="/var/lib/kubelet/pods/2670a524-52ac-4c55-9936-5873da1aa35d/volumes" Nov 03 02:02:15.575159 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.575131 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:02:15.627333 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.627276 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8ad5587-9637-45ea-9d12-97ba06c7011d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ad5587-9637-45ea-9d12-97ba06c7011d-istiod-ca-cert\") pod \"c8ad5587-9637-45ea-9d12-97ba06c7011d\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " Nov 03 02:02:15.627333 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.627318 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-data\") pod \"c8ad5587-9637-45ea-9d12-97ba06c7011d\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " Nov 03 02:02:15.627488 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.627388 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-envoy\") pod \"c8ad5587-9637-45ea-9d12-97ba06c7011d\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " Nov 03 02:02:15.627488 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.627428 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-workload-certs\") pod \"c8ad5587-9637-45ea-9d12-97ba06c7011d\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " Nov 03 02:02:15.627603 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.627496 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-token\" (UniqueName: \"kubernetes.io/projected/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-token\") pod \"c8ad5587-9637-45ea-9d12-97ba06c7011d\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " Nov 03 02:02:15.627603 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.627522 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-data" pod "c8ad5587-9637-45ea-9d12-97ba06c7011d" (UID: "c8ad5587-9637-45ea-9d12-97ba06c7011d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:15.627603 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.627551 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ad5587-9637-45ea-9d12-97ba06c7011d-kube-api-access-9r2ld\" (UniqueName: \"kubernetes.io/projected/c8ad5587-9637-45ea-9d12-97ba06c7011d-kube-api-access-9r2ld\") pod \"c8ad5587-9637-45ea-9d12-97ba06c7011d\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " Nov 03 02:02:15.627764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.627606 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-workload-socket\") pod \"c8ad5587-9637-45ea-9d12-97ba06c7011d\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " Nov 03 02:02:15.627764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.627610 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ad5587-9637-45ea-9d12-97ba06c7011d-istiod-ca-cert" pod "c8ad5587-9637-45ea-9d12-97ba06c7011d" (UID: "c8ad5587-9637-45ea-9d12-97ba06c7011d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:02:15.627764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.627652 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-podinfo\") pod \"c8ad5587-9637-45ea-9d12-97ba06c7011d\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " Nov 03 02:02:15.627764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.627671 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-credential-socket\") pod \"c8ad5587-9637-45ea-9d12-97ba06c7011d\" (UID: \"c8ad5587-9637-45ea-9d12-97ba06c7011d\") " Nov 03 02:02:15.627969 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.627773 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-workload-certs" pod "c8ad5587-9637-45ea-9d12-97ba06c7011d" (UID: "c8ad5587-9637-45ea-9d12-97ba06c7011d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:15.627969 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.627889 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-workload-socket" pod "c8ad5587-9637-45ea-9d12-97ba06c7011d" (UID: "c8ad5587-9637-45ea-9d12-97ba06c7011d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:15.628088 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.628064 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-credential-socket" pod "c8ad5587-9637-45ea-9d12-97ba06c7011d" (UID: "c8ad5587-9637-45ea-9d12-97ba06c7011d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:15.628245 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.628224 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:15.628324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.628250 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:15.628324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.628267 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:15.628324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.628282 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ad5587-9637-45ea-9d12-97ba06c7011d-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:15.628324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.628297 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:15.629655 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.629632 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-token" pod "c8ad5587-9637-45ea-9d12-97ba06c7011d" (UID: "c8ad5587-9637-45ea-9d12-97ba06c7011d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:15.629754 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.629678 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ad5587-9637-45ea-9d12-97ba06c7011d-kube-api-access-9r2ld" pod "c8ad5587-9637-45ea-9d12-97ba06c7011d" (UID: "c8ad5587-9637-45ea-9d12-97ba06c7011d"). InnerVolumeSpecName "kube-api-access-9r2ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:15.629817 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.629756 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-podinfo" pod "c8ad5587-9637-45ea-9d12-97ba06c7011d" (UID: "c8ad5587-9637-45ea-9d12-97ba06c7011d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:02:15.629870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.629826 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-envoy" pod "c8ad5587-9637-45ea-9d12-97ba06c7011d" (UID: "c8ad5587-9637-45ea-9d12-97ba06c7011d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:15.729615 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.729577 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:15.729615 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.729608 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:15.729615 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.729618 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ad5587-9637-45ea-9d12-97ba06c7011d-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:15.729905 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:15.729627 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r2ld\" (UniqueName: \"kubernetes.io/projected/c8ad5587-9637-45ea-9d12-97ba06c7011d-kube-api-access-9r2ld\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:16.042524 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.042485 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:16.098615 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.098582 5463 generic.go:356] "Generic (PLEG): container finished" podID="c8ad5587-9637-45ea-9d12-97ba06c7011d" containerID="c53f1bc2b17d01fc1368b544d60566b7a369ff799a32677107ce33b6a5fb7f0d" exitCode=0 Nov 03 02:02:16.098802 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.098649 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" Nov 03 02:02:16.098802 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.098671 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" event={"ID":"c8ad5587-9637-45ea-9d12-97ba06c7011d","Type":"ContainerDied","Data":"c53f1bc2b17d01fc1368b544d60566b7a369ff799a32677107ce33b6a5fb7f0d"} Nov 03 02:02:16.098802 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.098761 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt" event={"ID":"c8ad5587-9637-45ea-9d12-97ba06c7011d","Type":"ContainerDied","Data":"851d5a54c471135a14121a7a29372e5d86b1a722832ce0a0c20ac1b7f9724733"} Nov 03 02:02:16.098802 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.098791 5463 scope.go:122] "RemoveContainer" containerID="c53f1bc2b17d01fc1368b544d60566b7a369ff799a32677107ce33b6a5fb7f0d" Nov 03 02:02:16.109138 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.109118 5463 scope.go:122] "RemoveContainer" containerID="fca2a7791f245a50f1750f1851f30f177be70d6627c5db7e1fbd24d0f99898d4" Nov 03 02:02:16.117954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.117936 5463 scope.go:122] "RemoveContainer" containerID="c53f1bc2b17d01fc1368b544d60566b7a369ff799a32677107ce33b6a5fb7f0d" Nov 03 02:02:16.118251 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:02:16.118232 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c53f1bc2b17d01fc1368b544d60566b7a369ff799a32677107ce33b6a5fb7f0d\": container with ID starting with c53f1bc2b17d01fc1368b544d60566b7a369ff799a32677107ce33b6a5fb7f0d not found: ID does not exist" containerID="c53f1bc2b17d01fc1368b544d60566b7a369ff799a32677107ce33b6a5fb7f0d" Nov 03 02:02:16.118319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.118260 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c53f1bc2b17d01fc1368b544d60566b7a369ff799a32677107ce33b6a5fb7f0d"} err="failed to get container status \"c53f1bc2b17d01fc1368b544d60566b7a369ff799a32677107ce33b6a5fb7f0d\": rpc error: code = NotFound desc = could not find container \"c53f1bc2b17d01fc1368b544d60566b7a369ff799a32677107ce33b6a5fb7f0d\": container with ID starting with c53f1bc2b17d01fc1368b544d60566b7a369ff799a32677107ce33b6a5fb7f0d not found: ID does not exist" Nov 03 02:02:16.118319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.118277 5463 scope.go:122] "RemoveContainer" containerID="fca2a7791f245a50f1750f1851f30f177be70d6627c5db7e1fbd24d0f99898d4" Nov 03 02:02:16.118515 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:02:16.118497 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca2a7791f245a50f1750f1851f30f177be70d6627c5db7e1fbd24d0f99898d4\": container with ID starting with fca2a7791f245a50f1750f1851f30f177be70d6627c5db7e1fbd24d0f99898d4 not found: ID does not exist" containerID="fca2a7791f245a50f1750f1851f30f177be70d6627c5db7e1fbd24d0f99898d4" Nov 03 02:02:16.118567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.118518 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca2a7791f245a50f1750f1851f30f177be70d6627c5db7e1fbd24d0f99898d4"} err="failed to get container status \"fca2a7791f245a50f1750f1851f30f177be70d6627c5db7e1fbd24d0f99898d4\": rpc error: code = NotFound desc = could not find container \"fca2a7791f245a50f1750f1851f30f177be70d6627c5db7e1fbd24d0f99898d4\": container with ID starting with fca2a7791f245a50f1750f1851f30f177be70d6627c5db7e1fbd24d0f99898d4 not found: ID does not exist" Nov 03 02:02:16.210977 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.210947 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:16.266582 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.266549 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt"] Nov 03 02:02:16.315406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.315332 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6ac43c49-openshift-default-5c67dcd48c-n8tdt"] Nov 03 02:02:16.362546 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.362516 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:16.442824 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.442794 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8ad5587-9637-45ea-9d12-97ba06c7011d" path="/var/lib/kubelet/pods/c8ad5587-9637-45ea-9d12-97ba06c7011d/volumes" Nov 03 02:02:16.564180 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.564148 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89"] Nov 03 02:02:16.565991 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.565940 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7683f606-75x89"] Nov 03 02:02:16.728774 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.728742 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n"] Nov 03 02:02:16.729155 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.728957 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" podUID="02c00682-647d-4422-8ba4-1149c73b3eee" containerName="istio-proxy" containerID="cri-o://ffc6952367e996ae428f07f3f3c81f6f73c0ef5f4336eb62e471b13e4b928d02" gracePeriod=30 Nov 03 02:02:16.774782 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:16.774753 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:17.017796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.017758 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:17.190239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.190183 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:17.361687 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.361587 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:17.557246 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.557215 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:17.690096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.690065 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r"] Nov 03 02:02:17.690568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.690554 5463 cpu_manager.go:414] "containerMap: removing container" podUID="c8ad5587-9637-45ea-9d12-97ba06c7011d" containerName="istio-proxy" Nov 03 02:02:17.690615 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.690570 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8ad5587-9637-45ea-9d12-97ba06c7011d" containerName="istio-proxy" Nov 03 02:02:17.690615 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.690581 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2670a524-52ac-4c55-9936-5873da1aa35d" containerName="payload-processing" Nov 03 02:02:17.690615 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.690586 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2670a524-52ac-4c55-9936-5873da1aa35d" containerName="payload-processing" Nov 03 02:02:17.690615 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.690598 5463 cpu_manager.go:414] "containerMap: removing container" podUID="c8ad5587-9637-45ea-9d12-97ba06c7011d" containerName="istio-proxy" Nov 03 02:02:17.690615 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.690605 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8ad5587-9637-45ea-9d12-97ba06c7011d" containerName="istio-proxy" Nov 03 02:02:17.690766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.690621 5463 cpu_manager.go:414] "containerMap: removing container" podUID="0e391962-85cb-46bd-a13e-d53976cb0b8b" containerName="revoke-keys" Nov 03 02:02:17.690766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.690626 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e391962-85cb-46bd-a13e-d53976cb0b8b" containerName="revoke-keys" Nov 03 02:02:17.690766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.690694 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8ad5587-9637-45ea-9d12-97ba06c7011d" containerName="istio-proxy" Nov 03 02:02:17.690766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.690702 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2670a524-52ac-4c55-9936-5873da1aa35d" containerName="payload-processing" Nov 03 02:02:17.690766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.690708 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8ad5587-9637-45ea-9d12-97ba06c7011d" containerName="istio-proxy" Nov 03 02:02:17.690766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.690715 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e391962-85cb-46bd-a13e-d53976cb0b8b" containerName="revoke-keys" Nov 03 02:02:17.693780 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.693762 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r" Nov 03 02:02:17.746482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.746452 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:02:17.746905 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.746642 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r"] Nov 03 02:02:17.750532 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.750508 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdlp4\" (UniqueName: \"kubernetes.io/projected/0af47c99-d045-406c-8e5e-c1e3662f5058-kube-api-access-tdlp4\") pod \"maas-api-revoke-keys-e2e-ait-080015ba-6vj4r\" (UID: \"0af47c99-d045-406c-8e5e-c1e3662f5058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r" Nov 03 02:02:17.750660 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.750594 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af47c99-d045-406c-8e5e-c1e3662f5058-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-080015ba-6vj4r\" (UID: \"0af47c99-d045-406c-8e5e-c1e3662f5058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r" Nov 03 02:02:17.750702 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.750677 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af47c99-d045-406c-8e5e-c1e3662f5058-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-080015ba-6vj4r\" (UID: \"0af47c99-d045-406c-8e5e-c1e3662f5058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r" Nov 03 02:02:17.794607 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.794589 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:02:17.835742 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.835723 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:02:17.835944 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.835929 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:02:17.851955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.851933 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdlp4\" (UniqueName: \"kubernetes.io/projected/0af47c99-d045-406c-8e5e-c1e3662f5058-kube-api-access-tdlp4\") pod \"maas-api-revoke-keys-e2e-ait-080015ba-6vj4r\" (UID: \"0af47c99-d045-406c-8e5e-c1e3662f5058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r" Nov 03 02:02:17.852048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.851979 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af47c99-d045-406c-8e5e-c1e3662f5058-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-080015ba-6vj4r\" (UID: \"0af47c99-d045-406c-8e5e-c1e3662f5058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r" Nov 03 02:02:17.852048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.852001 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af47c99-d045-406c-8e5e-c1e3662f5058-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-080015ba-6vj4r\" (UID: \"0af47c99-d045-406c-8e5e-c1e3662f5058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r" Nov 03 02:02:17.852565 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.852548 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af47c99-d045-406c-8e5e-c1e3662f5058-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-080015ba-6vj4r\" (UID: \"0af47c99-d045-406c-8e5e-c1e3662f5058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r" Nov 03 02:02:17.854325 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.854308 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af47c99-d045-406c-8e5e-c1e3662f5058-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-080015ba-6vj4r\" (UID: \"0af47c99-d045-406c-8e5e-c1e3662f5058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r" Nov 03 02:02:17.908504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:17.908472 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdlp4\" (UniqueName: \"kubernetes.io/projected/0af47c99-d045-406c-8e5e-c1e3662f5058-kube-api-access-tdlp4\") pod \"maas-api-revoke-keys-e2e-ait-080015ba-6vj4r\" (UID: \"0af47c99-d045-406c-8e5e-c1e3662f5058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r" Nov 03 02:02:18.006707 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.006628 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r" Nov 03 02:02:18.034095 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.034065 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv"] Nov 03 02:02:18.037358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.037336 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.094502 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.094419 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a0e72e-openshift-default-dockercfg-cskrh\"" Nov 03 02:02:18.097220 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.097171 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv"] Nov 03 02:02:18.135141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.135112 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:18.154435 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.154409 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.154565 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.154456 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.154565 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.154532 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88462923-311f-4302-9503-b1dda3e244e2-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.154796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.154632 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88462923-311f-4302-9503-b1dda3e244e2-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.154796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.154668 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.154796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.154715 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88462923-311f-4302-9503-b1dda3e244e2-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.154796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.154753 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l8n4\" (UniqueName: \"kubernetes.io/projected/88462923-311f-4302-9503-b1dda3e244e2-kube-api-access-8l8n4\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.154796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.154781 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.155025 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.154831 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.256304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.256268 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88462923-311f-4302-9503-b1dda3e244e2-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.256304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.256311 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.256553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.256342 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88462923-311f-4302-9503-b1dda3e244e2-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.256553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.256369 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l8n4\" (UniqueName: \"kubernetes.io/projected/88462923-311f-4302-9503-b1dda3e244e2-kube-api-access-8l8n4\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.256553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.256389 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.256553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.256426 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.256553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.256475 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.256553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.256508 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.256880 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.256830 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.256953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.256872 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88462923-311f-4302-9503-b1dda3e244e2-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.256953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.256896 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.256953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.256944 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.257110 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.257007 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.257466 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.257438 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88462923-311f-4302-9503-b1dda3e244e2-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.258692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.258672 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.259044 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.259028 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88462923-311f-4302-9503-b1dda3e244e2-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.313210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.313173 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88462923-311f-4302-9503-b1dda3e244e2-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.313436 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.313417 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l8n4\" (UniqueName: \"kubernetes.io/projected/88462923-311f-4302-9503-b1dda3e244e2-kube-api-access-8l8n4\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.351236 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.351179 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:18.354644 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.354621 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:18.376957 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:02:18.376458 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af47c99_d045_406c_8e5e_c1e3662f5058.slice/crio-72d4e1f67855603ad99bc619cc587bb992bbd1bd174172f320a76acc36de30c6 WatchSource:0}: Error finding container 72d4e1f67855603ad99bc619cc587bb992bbd1bd174172f320a76acc36de30c6: Status 404 returned error can't find the container with id 72d4e1f67855603ad99bc619cc587bb992bbd1bd174172f320a76acc36de30c6 Nov 03 02:02:18.380128 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.380087 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r"] Nov 03 02:02:18.453250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.452152 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e391962-85cb-46bd-a13e-d53976cb0b8b" path="/var/lib/kubelet/pods/0e391962-85cb-46bd-a13e-d53976cb0b8b/volumes" Nov 03 02:02:18.731613 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:02:18.731584 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88462923_311f_4302_9503_b1dda3e244e2.slice/crio-1b92c78ca485c5cb0a648dd77fc5ebccd41a9308a8e37e3e65b9fe9917597633 WatchSource:0}: Error finding container 1b92c78ca485c5cb0a648dd77fc5ebccd41a9308a8e37e3e65b9fe9917597633: Status 404 returned error can't find the container with id 1b92c78ca485c5cb0a648dd77fc5ebccd41a9308a8e37e3e65b9fe9917597633 Nov 03 02:02:18.731975 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.731858 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv"] Nov 03 02:02:18.733762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.733720 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:18.733857 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.733806 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:18.733857 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:18.733848 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:19.119550 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:19.119448 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r" event={"ID":"0af47c99-d045-406c-8e5e-c1e3662f5058","Type":"ContainerStarted","Data":"50453e98c6dcffb1d644880d61d215b9360a9a348db198c212e3f95123964ad6"} Nov 03 02:02:19.119988 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:19.119553 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r" event={"ID":"0af47c99-d045-406c-8e5e-c1e3662f5058","Type":"ContainerStarted","Data":"72d4e1f67855603ad99bc619cc587bb992bbd1bd174172f320a76acc36de30c6"} Nov 03 02:02:19.121728 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:19.121696 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" event={"ID":"88462923-311f-4302-9503-b1dda3e244e2","Type":"ContainerStarted","Data":"fc5953001997a0d8392b007654bb435c0fbd39fad5bdfa6f8ce6b6f723276eae"} Nov 03 02:02:19.121845 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:19.121760 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" event={"ID":"88462923-311f-4302-9503-b1dda3e244e2","Type":"ContainerStarted","Data":"1b92c78ca485c5cb0a648dd77fc5ebccd41a9308a8e37e3e65b9fe9917597633"} Nov 03 02:02:19.235529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:19.235486 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r" podStartSLOduration=2.235472282 podStartE2EDuration="2.235472282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:19.23434204 +0000 UTC m=+620.918451192" watchObservedRunningTime="2026-11-03 02:02:19.235472282 +0000 UTC m=+620.919581436" Nov 03 02:02:19.352215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:19.352177 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:19.352533 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:19.352491 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" podStartSLOduration=2.352478155 podStartE2EDuration="2.352478155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:19.349773842 +0000 UTC m=+621.033882998" watchObservedRunningTime="2026-11-03 02:02:19.352478155 +0000 UTC m=+621.036587395" Nov 03 02:02:19.358014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:19.357981 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:20.128893 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:20.128860 5463 generic.go:356] "Generic (PLEG): container finished" podID="0af47c99-d045-406c-8e5e-c1e3662f5058" containerID="50453e98c6dcffb1d644880d61d215b9360a9a348db198c212e3f95123964ad6" exitCode=0 Nov 03 02:02:20.129309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:20.128946 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r" event={"ID":"0af47c99-d045-406c-8e5e-c1e3662f5058","Type":"ContainerDied","Data":"50453e98c6dcffb1d644880d61d215b9360a9a348db198c212e3f95123964ad6"} Nov 03 02:02:20.129309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:20.129228 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:20.130077 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:20.130060 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:02:21.076503 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:21.076471 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:21.271789 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:21.271766 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r" Nov 03 02:02:21.395283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:21.395261 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0af47c99-d045-406c-8e5e-c1e3662f5058-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af47c99-d045-406c-8e5e-c1e3662f5058-maas-api-openshift-ca\") pod \"0af47c99-d045-406c-8e5e-c1e3662f5058\" (UID: \"0af47c99-d045-406c-8e5e-c1e3662f5058\") " Nov 03 02:02:21.395435 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:21.395320 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0af47c99-d045-406c-8e5e-c1e3662f5058-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af47c99-d045-406c-8e5e-c1e3662f5058-maas-api-xks-ca\") pod \"0af47c99-d045-406c-8e5e-c1e3662f5058\" (UID: \"0af47c99-d045-406c-8e5e-c1e3662f5058\") " Nov 03 02:02:21.395435 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:21.395370 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af47c99-d045-406c-8e5e-c1e3662f5058-kube-api-access-tdlp4\" (UniqueName: \"kubernetes.io/projected/0af47c99-d045-406c-8e5e-c1e3662f5058-kube-api-access-tdlp4\") pod \"0af47c99-d045-406c-8e5e-c1e3662f5058\" (UID: \"0af47c99-d045-406c-8e5e-c1e3662f5058\") " Nov 03 02:02:21.395709 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:21.395671 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af47c99-d045-406c-8e5e-c1e3662f5058-maas-api-openshift-ca" pod "0af47c99-d045-406c-8e5e-c1e3662f5058" (UID: "0af47c99-d045-406c-8e5e-c1e3662f5058"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:02:21.395888 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:21.395866 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af47c99-d045-406c-8e5e-c1e3662f5058-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:21.397483 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:21.397458 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af47c99-d045-406c-8e5e-c1e3662f5058-maas-api-xks-ca" pod "0af47c99-d045-406c-8e5e-c1e3662f5058" (UID: "0af47c99-d045-406c-8e5e-c1e3662f5058"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:02:21.397570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:21.397509 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af47c99-d045-406c-8e5e-c1e3662f5058-kube-api-access-tdlp4" pod "0af47c99-d045-406c-8e5e-c1e3662f5058" (UID: "0af47c99-d045-406c-8e5e-c1e3662f5058"). InnerVolumeSpecName "kube-api-access-tdlp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:21.496684 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:21.496659 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af47c99-d045-406c-8e5e-c1e3662f5058-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:21.496684 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:21.496682 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdlp4\" (UniqueName: \"kubernetes.io/projected/0af47c99-d045-406c-8e5e-c1e3662f5058-kube-api-access-tdlp4\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:21.752230 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:21.752159 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:21.928530 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:21.928498 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:21.988916 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:21.988891 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:02:22.102425 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.102333 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c00682-647d-4422-8ba4-1149c73b3eee-istio-token\" (UniqueName: \"kubernetes.io/projected/02c00682-647d-4422-8ba4-1149c73b3eee-istio-token\") pod \"02c00682-647d-4422-8ba4-1149c73b3eee\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " Nov 03 02:02:22.102425 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.102398 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-workload-certs\") pod \"02c00682-647d-4422-8ba4-1149c73b3eee\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " Nov 03 02:02:22.102644 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.102432 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02c00682-647d-4422-8ba4-1149c73b3eee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c00682-647d-4422-8ba4-1149c73b3eee-istiod-ca-cert\") pod \"02c00682-647d-4422-8ba4-1149c73b3eee\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " Nov 03 02:02:22.102644 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.102483 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02c00682-647d-4422-8ba4-1149c73b3eee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c00682-647d-4422-8ba4-1149c73b3eee-istio-podinfo\") pod \"02c00682-647d-4422-8ba4-1149c73b3eee\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " Nov 03 02:02:22.102644 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.102513 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c00682-647d-4422-8ba4-1149c73b3eee-kube-api-access-nthhf\" (UniqueName: \"kubernetes.io/projected/02c00682-647d-4422-8ba4-1149c73b3eee-kube-api-access-nthhf\") pod \"02c00682-647d-4422-8ba4-1149c73b3eee\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " Nov 03 02:02:22.102813 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.102605 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-credential-socket\") pod \"02c00682-647d-4422-8ba4-1149c73b3eee\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " Nov 03 02:02:22.102897 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.102873 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c00682-647d-4422-8ba4-1149c73b3eee-istiod-ca-cert" pod "02c00682-647d-4422-8ba4-1149c73b3eee" (UID: "02c00682-647d-4422-8ba4-1149c73b3eee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:02:22.103052 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.102968 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-istio-envoy\") pod \"02c00682-647d-4422-8ba4-1149c73b3eee\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " Nov 03 02:02:22.103276 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.103255 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-istio-data\") pod \"02c00682-647d-4422-8ba4-1149c73b3eee\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " Nov 03 02:02:22.103468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.103442 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-workload-socket\") pod \"02c00682-647d-4422-8ba4-1149c73b3eee\" (UID: \"02c00682-647d-4422-8ba4-1149c73b3eee\") " Nov 03 02:02:22.103666 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.103068 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-workload-certs" pod "02c00682-647d-4422-8ba4-1149c73b3eee" (UID: "02c00682-647d-4422-8ba4-1149c73b3eee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:22.103666 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.103360 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-credential-socket" pod "02c00682-647d-4422-8ba4-1149c73b3eee" (UID: "02c00682-647d-4422-8ba4-1149c73b3eee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:22.103666 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.103485 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-istio-data" pod "02c00682-647d-4422-8ba4-1149c73b3eee" (UID: "02c00682-647d-4422-8ba4-1149c73b3eee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:22.103842 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.103686 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-workload-socket" pod "02c00682-647d-4422-8ba4-1149c73b3eee" (UID: "02c00682-647d-4422-8ba4-1149c73b3eee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:22.104415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.104395 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:22.104538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.104511 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:22.104538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.104537 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:22.104792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.104549 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c00682-647d-4422-8ba4-1149c73b3eee-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:22.104792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.104562 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:22.108156 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.107893 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02c00682-647d-4422-8ba4-1149c73b3eee-istio-podinfo" pod "02c00682-647d-4422-8ba4-1149c73b3eee" (UID: "02c00682-647d-4422-8ba4-1149c73b3eee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:02:22.108156 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.107893 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c00682-647d-4422-8ba4-1149c73b3eee-istio-token" pod "02c00682-647d-4422-8ba4-1149c73b3eee" (UID: "02c00682-647d-4422-8ba4-1149c73b3eee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:22.108156 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.107953 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-istio-envoy" pod "02c00682-647d-4422-8ba4-1149c73b3eee" (UID: "02c00682-647d-4422-8ba4-1149c73b3eee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:22.108156 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.107940 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c00682-647d-4422-8ba4-1149c73b3eee-kube-api-access-nthhf" pod "02c00682-647d-4422-8ba4-1149c73b3eee" (UID: "02c00682-647d-4422-8ba4-1149c73b3eee"). InnerVolumeSpecName "kube-api-access-nthhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:22.145330 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.145143 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r" Nov 03 02:02:22.145330 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.145186 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r" event={"ID":"0af47c99-d045-406c-8e5e-c1e3662f5058","Type":"ContainerDied","Data":"72d4e1f67855603ad99bc619cc587bb992bbd1bd174172f320a76acc36de30c6"} Nov 03 02:02:22.145330 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.145302 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72d4e1f67855603ad99bc619cc587bb992bbd1bd174172f320a76acc36de30c6" Nov 03 02:02:22.147053 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.147024 5463 generic.go:356] "Generic (PLEG): container finished" podID="02c00682-647d-4422-8ba4-1149c73b3eee" containerID="ffc6952367e996ae428f07f3f3c81f6f73c0ef5f4336eb62e471b13e4b928d02" exitCode=0 Nov 03 02:02:22.147288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.147076 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" Nov 03 02:02:22.147288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.147112 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" event={"ID":"02c00682-647d-4422-8ba4-1149c73b3eee","Type":"ContainerDied","Data":"ffc6952367e996ae428f07f3f3c81f6f73c0ef5f4336eb62e471b13e4b928d02"} Nov 03 02:02:22.147288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.147155 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n" event={"ID":"02c00682-647d-4422-8ba4-1149c73b3eee","Type":"ContainerDied","Data":"823f53bc8557dd06e3bd0eadb75cbf1743e61dba2610588e744c3604ce7ce939"} Nov 03 02:02:22.147288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.147177 5463 scope.go:122] "RemoveContainer" containerID="ffc6952367e996ae428f07f3f3c81f6f73c0ef5f4336eb62e471b13e4b928d02" Nov 03 02:02:22.160515 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.160466 5463 scope.go:122] "RemoveContainer" containerID="389848060def630d22ec439f92e6be6ffe483fef0c732ccd5aafd2e854f82832" Nov 03 02:02:22.161615 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.161071 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn"] Nov 03 02:02:22.161756 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.161643 5463 cpu_manager.go:414] "containerMap: removing container" podUID="0af47c99-d045-406c-8e5e-c1e3662f5058" containerName="revoke-keys" Nov 03 02:02:22.161756 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.161657 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af47c99-d045-406c-8e5e-c1e3662f5058" containerName="revoke-keys" Nov 03 02:02:22.161756 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.161675 5463 cpu_manager.go:414] "containerMap: removing container" podUID="02c00682-647d-4422-8ba4-1149c73b3eee" containerName="istio-proxy" Nov 03 02:02:22.161756 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.161680 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c00682-647d-4422-8ba4-1149c73b3eee" containerName="istio-proxy" Nov 03 02:02:22.161756 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.161698 5463 cpu_manager.go:414] "containerMap: removing container" podUID="02c00682-647d-4422-8ba4-1149c73b3eee" containerName="istio-proxy" Nov 03 02:02:22.161756 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.161703 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c00682-647d-4422-8ba4-1149c73b3eee" containerName="istio-proxy" Nov 03 02:02:22.162008 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.161769 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c00682-647d-4422-8ba4-1149c73b3eee" containerName="istio-proxy" Nov 03 02:02:22.162008 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.161781 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0af47c99-d045-406c-8e5e-c1e3662f5058" containerName="revoke-keys" Nov 03 02:02:22.166168 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.166150 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn" Nov 03 02:02:22.170400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.170380 5463 scope.go:122] "RemoveContainer" containerID="ffc6952367e996ae428f07f3f3c81f6f73c0ef5f4336eb62e471b13e4b928d02" Nov 03 02:02:22.170720 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:02:22.170692 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffc6952367e996ae428f07f3f3c81f6f73c0ef5f4336eb62e471b13e4b928d02\": container with ID starting with ffc6952367e996ae428f07f3f3c81f6f73c0ef5f4336eb62e471b13e4b928d02 not found: ID does not exist" containerID="ffc6952367e996ae428f07f3f3c81f6f73c0ef5f4336eb62e471b13e4b928d02" Nov 03 02:02:22.170808 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.170723 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffc6952367e996ae428f07f3f3c81f6f73c0ef5f4336eb62e471b13e4b928d02"} err="failed to get container status \"ffc6952367e996ae428f07f3f3c81f6f73c0ef5f4336eb62e471b13e4b928d02\": rpc error: code = NotFound desc = could not find container \"ffc6952367e996ae428f07f3f3c81f6f73c0ef5f4336eb62e471b13e4b928d02\": container with ID starting with ffc6952367e996ae428f07f3f3c81f6f73c0ef5f4336eb62e471b13e4b928d02 not found: ID does not exist" Nov 03 02:02:22.170808 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.170746 5463 scope.go:122] "RemoveContainer" containerID="389848060def630d22ec439f92e6be6ffe483fef0c732ccd5aafd2e854f82832" Nov 03 02:02:22.171021 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:02:22.171003 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"389848060def630d22ec439f92e6be6ffe483fef0c732ccd5aafd2e854f82832\": container with ID starting with 389848060def630d22ec439f92e6be6ffe483fef0c732ccd5aafd2e854f82832 not found: ID does not exist" containerID="389848060def630d22ec439f92e6be6ffe483fef0c732ccd5aafd2e854f82832" Nov 03 02:02:22.171090 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.171028 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"389848060def630d22ec439f92e6be6ffe483fef0c732ccd5aafd2e854f82832"} err="failed to get container status \"389848060def630d22ec439f92e6be6ffe483fef0c732ccd5aafd2e854f82832\": rpc error: code = NotFound desc = could not find container \"389848060def630d22ec439f92e6be6ffe483fef0c732ccd5aafd2e854f82832\": container with ID starting with 389848060def630d22ec439f92e6be6ffe483fef0c732ccd5aafd2e854f82832 not found: ID does not exist" Nov 03 02:02:22.206145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.205770 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c00682-647d-4422-8ba4-1149c73b3eee-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:22.206145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.205799 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c00682-647d-4422-8ba4-1149c73b3eee-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:22.206145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.205814 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c00682-647d-4422-8ba4-1149c73b3eee-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:22.206145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.205829 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nthhf\" (UniqueName: \"kubernetes.io/projected/02c00682-647d-4422-8ba4-1149c73b3eee-kube-api-access-nthhf\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:22.206995 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.206972 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn"] Nov 03 02:02:22.232214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.230542 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-521f1f-dockercfg-lgfsk\"" Nov 03 02:02:22.232214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.230632 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-521f1f\"" Nov 03 02:02:22.293291 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.293263 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c"] Nov 03 02:02:22.294346 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.294327 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c00682-647d-4422-8ba4-1149c73b3eee" containerName="istio-proxy" Nov 03 02:02:22.296735 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.296717 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c" Nov 03 02:02:22.306520 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.306501 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2d8c\" (UniqueName: \"kubernetes.io/projected/462caa1d-f10d-4355-ac46-cf9ee0153d23-kube-api-access-t2d8c\") pod \"payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn\" (UID: \"462caa1d-f10d-4355-ac46-cf9ee0153d23\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn" Nov 03 02:02:22.306647 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.306632 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462caa1d-f10d-4355-ac46-cf9ee0153d23-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn\" (UID: \"462caa1d-f10d-4355-ac46-cf9ee0153d23\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn" Nov 03 02:02:22.314240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.314186 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n"] Nov 03 02:02:22.319914 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.319894 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-7683f606-gw-openshift-default-575f8c96df-rmf5n"] Nov 03 02:02:22.348908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.348879 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c"] Nov 03 02:02:22.407398 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.407366 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2d8c\" (UniqueName: \"kubernetes.io/projected/462caa1d-f10d-4355-ac46-cf9ee0153d23-kube-api-access-t2d8c\") pod \"payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn\" (UID: \"462caa1d-f10d-4355-ac46-cf9ee0153d23\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn" Nov 03 02:02:22.407568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.407495 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhsfw\" (UniqueName: \"kubernetes.io/projected/6c5f16c6-739d-4bc0-af05-4587087cc0be-kube-api-access-lhsfw\") pod \"payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c\" (UID: \"6c5f16c6-739d-4bc0-af05-4587087cc0be\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c" Nov 03 02:02:22.407568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.407562 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462caa1d-f10d-4355-ac46-cf9ee0153d23-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn\" (UID: \"462caa1d-f10d-4355-ac46-cf9ee0153d23\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn" Nov 03 02:02:22.407667 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.407622 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c5f16c6-739d-4bc0-af05-4587087cc0be-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c\" (UID: \"6c5f16c6-739d-4bc0-af05-4587087cc0be\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c" Nov 03 02:02:22.408335 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.408319 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462caa1d-f10d-4355-ac46-cf9ee0153d23-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn\" (UID: \"462caa1d-f10d-4355-ac46-cf9ee0153d23\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn" Nov 03 02:02:22.442126 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.442102 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02c00682-647d-4422-8ba4-1149c73b3eee" path="/var/lib/kubelet/pods/02c00682-647d-4422-8ba4-1149c73b3eee/volumes" Nov 03 02:02:22.465047 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.465019 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2d8c\" (UniqueName: \"kubernetes.io/projected/462caa1d-f10d-4355-ac46-cf9ee0153d23-kube-api-access-t2d8c\") pod \"payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn\" (UID: \"462caa1d-f10d-4355-ac46-cf9ee0153d23\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn" Nov 03 02:02:22.480896 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.480877 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn" Nov 03 02:02:22.509967 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.509928 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhsfw\" (UniqueName: \"kubernetes.io/projected/6c5f16c6-739d-4bc0-af05-4587087cc0be-kube-api-access-lhsfw\") pod \"payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c\" (UID: \"6c5f16c6-739d-4bc0-af05-4587087cc0be\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c" Nov 03 02:02:22.510104 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.510054 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c5f16c6-739d-4bc0-af05-4587087cc0be-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c\" (UID: \"6c5f16c6-739d-4bc0-af05-4587087cc0be\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c" Nov 03 02:02:22.510821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.510799 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c5f16c6-739d-4bc0-af05-4587087cc0be-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c\" (UID: \"6c5f16c6-739d-4bc0-af05-4587087cc0be\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c" Nov 03 02:02:22.566753 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.566728 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhsfw\" (UniqueName: \"kubernetes.io/projected/6c5f16c6-739d-4bc0-af05-4587087cc0be-kube-api-access-lhsfw\") pod \"payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c\" (UID: \"6c5f16c6-739d-4bc0-af05-4587087cc0be\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c" Nov 03 02:02:22.610804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.610777 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c" Nov 03 02:02:22.844664 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:02:22.844630 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462caa1d_f10d_4355_ac46_cf9ee0153d23.slice/crio-822c8ccac0f14308c7d575f988788b1b9ce96be99a71c4be33ade849b7980f86 WatchSource:0}: Error finding container 822c8ccac0f14308c7d575f988788b1b9ce96be99a71c4be33ade849b7980f86: Status 404 returned error can't find the container with id 822c8ccac0f14308c7d575f988788b1b9ce96be99a71c4be33ade849b7980f86 Nov 03 02:02:22.848784 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.848759 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn"] Nov 03 02:02:22.883510 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.883488 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h"] Nov 03 02:02:22.989174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.989142 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c"] Nov 03 02:02:22.992383 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:22.992098 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m"] Nov 03 02:02:23.154543 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:23.154503 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c" event={"ID":"6c5f16c6-739d-4bc0-af05-4587087cc0be","Type":"ContainerStarted","Data":"6996ee59b386b56462e63ecb8f0545da95506e59f9b4cf95565eba49e7cb8190"} Nov 03 02:02:23.154726 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:23.154562 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c" event={"ID":"6c5f16c6-739d-4bc0-af05-4587087cc0be","Type":"ContainerStarted","Data":"40001a729609402088f8af92019fc94b13f96831f5ea7452a3037e750762dd03"} Nov 03 02:02:23.154726 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:23.154692 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c" Nov 03 02:02:23.157205 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:23.157175 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn" event={"ID":"462caa1d-f10d-4355-ac46-cf9ee0153d23","Type":"ContainerStarted","Data":"8458c21e0225ece4dfd00c018dbe6aeac775d6d6970438db5b4c3de8c05df006"} Nov 03 02:02:23.157319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:23.157244 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn" event={"ID":"462caa1d-f10d-4355-ac46-cf9ee0153d23","Type":"ContainerStarted","Data":"822c8ccac0f14308c7d575f988788b1b9ce96be99a71c4be33ade849b7980f86"} Nov 03 02:02:23.157319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:23.157268 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn" Nov 03 02:02:23.270026 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:23.269947 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:23.272797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:23.272748 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c" podStartSLOduration=1.272732944 podStartE2EDuration="1.272732944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:23.271740615 +0000 UTC m=+624.955849782" watchObservedRunningTime="2026-11-03 02:02:23.272732944 +0000 UTC m=+624.956842100" Nov 03 02:02:23.389636 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:23.389581 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn" podStartSLOduration=1.389565661 podStartE2EDuration="1.389565661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:23.387752316 +0000 UTC m=+625.071861472" watchObservedRunningTime="2026-11-03 02:02:23.389565661 +0000 UTC m=+625.073674834" Nov 03 02:02:23.421614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:23.421584 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:23.610364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:23.610287 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:23.774269 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:23.774228 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:24.019889 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:24.019860 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:25.678012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:25.677976 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:27.291397 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:27.291363 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:27.603266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:27.603171 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:27.749468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:27.749425 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:27.885119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:27.885031 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h" podUID="86377347-5a8a-4be2-ba1d-a333b78c57ec" containerName="payload-processing" containerID="cri-o://3d325b4a4f9bae73e7768bd0e3d10a22fc244bccd145ae472d59873e89e1dd9d" gracePeriod=25 Nov 03 02:02:27.971795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:27.971751 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj"] Nov 03 02:02:27.971969 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:27.971949 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" podUID="9c143c15-d9ea-4a97-81ed-04fa713fc5d7" containerName="istio-proxy" containerID="cri-o://21f72345861d4eb204999c98f479b7fd0eaa645e91a31b9e219507253d32cd9e" gracePeriod=30 Nov 03 02:02:27.989590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:27.989562 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:27.993340 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:27.993313 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m" podUID="b0a0798f-c015-4dc1-b468-c764722cb2ff" containerName="payload-pre-processing" containerID="cri-o://57158d80ac6b7042e10b45daba76d600dbb0aaac38f07ecd1c21075eda39d0ff" gracePeriod=25 Nov 03 02:02:28.193421 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.193142 5463 generic.go:356] "Generic (PLEG): container finished" podID="86377347-5a8a-4be2-ba1d-a333b78c57ec" containerID="3d325b4a4f9bae73e7768bd0e3d10a22fc244bccd145ae472d59873e89e1dd9d" exitCode=0 Nov 03 02:02:28.193421 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.193248 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h" event={"ID":"86377347-5a8a-4be2-ba1d-a333b78c57ec","Type":"ContainerDied","Data":"3d325b4a4f9bae73e7768bd0e3d10a22fc244bccd145ae472d59873e89e1dd9d"} Nov 03 02:02:28.193421 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.193354 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h" event={"ID":"86377347-5a8a-4be2-ba1d-a333b78c57ec","Type":"ContainerDied","Data":"d8b851624f1cc63a2a2ddfbe65685aa28b10de7677b767aa913fd9bb55be1e3a"} Nov 03 02:02:28.193421 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.193384 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8b851624f1cc63a2a2ddfbe65685aa28b10de7677b767aa913fd9bb55be1e3a" Nov 03 02:02:28.195347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.195324 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h" Nov 03 02:02:28.196044 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.196009 5463 generic.go:356] "Generic (PLEG): container finished" podID="b0a0798f-c015-4dc1-b468-c764722cb2ff" containerID="57158d80ac6b7042e10b45daba76d600dbb0aaac38f07ecd1c21075eda39d0ff" exitCode=0 Nov 03 02:02:28.196158 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.196052 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m" event={"ID":"b0a0798f-c015-4dc1-b468-c764722cb2ff","Type":"ContainerDied","Data":"57158d80ac6b7042e10b45daba76d600dbb0aaac38f07ecd1c21075eda39d0ff"} Nov 03 02:02:28.247164 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.247144 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m" Nov 03 02:02:28.286546 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.286513 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86377347-5a8a-4be2-ba1d-a333b78c57ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86377347-5a8a-4be2-ba1d-a333b78c57ec-plugins-config-volume\") pod \"86377347-5a8a-4be2-ba1d-a333b78c57ec\" (UID: \"86377347-5a8a-4be2-ba1d-a333b78c57ec\") " Nov 03 02:02:28.286682 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.286648 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86377347-5a8a-4be2-ba1d-a333b78c57ec-kube-api-access-wt94t\" (UniqueName: \"kubernetes.io/projected/86377347-5a8a-4be2-ba1d-a333b78c57ec-kube-api-access-wt94t\") pod \"86377347-5a8a-4be2-ba1d-a333b78c57ec\" (UID: \"86377347-5a8a-4be2-ba1d-a333b78c57ec\") " Nov 03 02:02:28.286925 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.286902 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86377347-5a8a-4be2-ba1d-a333b78c57ec-plugins-config-volume" pod "86377347-5a8a-4be2-ba1d-a333b78c57ec" (UID: "86377347-5a8a-4be2-ba1d-a333b78c57ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:02:28.287025 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.287010 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86377347-5a8a-4be2-ba1d-a333b78c57ec-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:28.288689 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.288664 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86377347-5a8a-4be2-ba1d-a333b78c57ec-kube-api-access-wt94t" pod "86377347-5a8a-4be2-ba1d-a333b78c57ec" (UID: "86377347-5a8a-4be2-ba1d-a333b78c57ec"). InnerVolumeSpecName "kube-api-access-wt94t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:28.387602 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.387570 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0a0798f-c015-4dc1-b468-c764722cb2ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a0798f-c015-4dc1-b468-c764722cb2ff-plugins-config-volume\") pod \"b0a0798f-c015-4dc1-b468-c764722cb2ff\" (UID: \"b0a0798f-c015-4dc1-b468-c764722cb2ff\") " Nov 03 02:02:28.387982 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.387658 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a0798f-c015-4dc1-b468-c764722cb2ff-kube-api-access-mdm8h\" (UniqueName: \"kubernetes.io/projected/b0a0798f-c015-4dc1-b468-c764722cb2ff-kube-api-access-mdm8h\") pod \"b0a0798f-c015-4dc1-b468-c764722cb2ff\" (UID: \"b0a0798f-c015-4dc1-b468-c764722cb2ff\") " Nov 03 02:02:28.387982 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.387865 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt94t\" (UniqueName: \"kubernetes.io/projected/86377347-5a8a-4be2-ba1d-a333b78c57ec-kube-api-access-wt94t\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:28.387982 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.387911 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a0798f-c015-4dc1-b468-c764722cb2ff-plugins-config-volume" pod "b0a0798f-c015-4dc1-b468-c764722cb2ff" (UID: "b0a0798f-c015-4dc1-b468-c764722cb2ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:02:28.389676 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.389653 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a0798f-c015-4dc1-b468-c764722cb2ff-kube-api-access-mdm8h" pod "b0a0798f-c015-4dc1-b468-c764722cb2ff" (UID: "b0a0798f-c015-4dc1-b468-c764722cb2ff"). InnerVolumeSpecName "kube-api-access-mdm8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:28.489352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.489311 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a0798f-c015-4dc1-b468-c764722cb2ff-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:28.489352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.489350 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdm8h\" (UniqueName: \"kubernetes.io/projected/b0a0798f-c015-4dc1-b468-c764722cb2ff-kube-api-access-mdm8h\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:28.829561 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:28.829476 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:29.021276 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.021235 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:29.194744 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.194688 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:29.204774 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.204740 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m" event={"ID":"b0a0798f-c015-4dc1-b468-c764722cb2ff","Type":"ContainerDied","Data":"7d1be1587326b094e7f2fc5c7d8cb3c61243f6e7c035919aec13804fce474c72"} Nov 03 02:02:29.204774 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.204761 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h" Nov 03 02:02:29.204969 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.204758 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m" Nov 03 02:02:29.204969 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.204816 5463 scope.go:122] "RemoveContainer" containerID="57158d80ac6b7042e10b45daba76d600dbb0aaac38f07ecd1c21075eda39d0ff" Nov 03 02:02:29.322668 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.322632 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24"] Nov 03 02:02:29.323119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.323105 5463 cpu_manager.go:414] "containerMap: removing container" podUID="86377347-5a8a-4be2-ba1d-a333b78c57ec" containerName="payload-processing" Nov 03 02:02:29.323169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.323120 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86377347-5a8a-4be2-ba1d-a333b78c57ec" containerName="payload-processing" Nov 03 02:02:29.323169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.323152 5463 cpu_manager.go:414] "containerMap: removing container" podUID="b0a0798f-c015-4dc1-b468-c764722cb2ff" containerName="payload-pre-processing" Nov 03 02:02:29.323169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.323158 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a0798f-c015-4dc1-b468-c764722cb2ff" containerName="payload-pre-processing" Nov 03 02:02:29.323286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.323274 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="86377347-5a8a-4be2-ba1d-a333b78c57ec" containerName="payload-processing" Nov 03 02:02:29.323318 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.323293 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a0798f-c015-4dc1-b468-c764722cb2ff" containerName="payload-pre-processing" Nov 03 02:02:29.326513 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.326498 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.378995 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.378964 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h"] Nov 03 02:02:29.379915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.379888 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-1bccbd-openshift-default-dockercfg-c57w7\"" Nov 03 02:02:29.382407 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.382366 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24"] Nov 03 02:02:29.385970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.385946 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fbf24ed8-bcf9f8c9c-fnw6h"] Nov 03 02:02:29.407237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.407207 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88"] Nov 03 02:02:29.410635 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.410604 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.410635 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.410619 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88" Nov 03 02:02:29.410830 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.410719 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/831449be-effc-4efe-8005-f53a501168a1-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.410830 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.410785 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.410830 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.410809 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/831449be-effc-4efe-8005-f53a501168a1-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.410981 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.410916 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.410981 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.410957 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76sw2\" (UniqueName: \"kubernetes.io/projected/831449be-effc-4efe-8005-f53a501168a1-kube-api-access-76sw2\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.411162 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.411078 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/831449be-effc-4efe-8005-f53a501168a1-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.411162 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.411120 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.411316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.411221 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.417055 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.417028 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:29.464096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.464032 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-a0e72e-dockercfg-4p25h\"" Nov 03 02:02:29.480089 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.480040 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88"] Nov 03 02:02:29.491302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.491280 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m"] Nov 03 02:02:29.502621 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.502601 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fbf24ed8-5fd8f7cb47-sf45m"] Nov 03 02:02:29.503784 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.503768 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-a0e72e\"" Nov 03 02:02:29.512319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.512295 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsnfx\" (UniqueName: \"kubernetes.io/projected/9b7b703f-6c31-4881-8e71-a23b27c3e6e7-kube-api-access-xsnfx\") pod \"payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88\" (UID: \"9b7b703f-6c31-4881-8e71-a23b27c3e6e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88" Nov 03 02:02:29.512400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.512350 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/831449be-effc-4efe-8005-f53a501168a1-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.512437 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.512404 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.512504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.512434 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.512504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.512457 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.512504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.512481 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7b703f-6c31-4881-8e71-a23b27c3e6e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88\" (UID: \"9b7b703f-6c31-4881-8e71-a23b27c3e6e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88" Nov 03 02:02:29.512655 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.512569 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/831449be-effc-4efe-8005-f53a501168a1-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.512835 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.512730 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.513027 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.513012 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.513083 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.513007 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.513083 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.512762 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/831449be-effc-4efe-8005-f53a501168a1-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.513218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.513123 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.513218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.513145 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76sw2\" (UniqueName: \"kubernetes.io/projected/831449be-effc-4efe-8005-f53a501168a1-kube-api-access-76sw2\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.513499 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.513264 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.513499 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.513411 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.513670 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.513652 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/831449be-effc-4efe-8005-f53a501168a1-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.514929 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.514908 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/831449be-effc-4efe-8005-f53a501168a1-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.515414 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.515393 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.574918 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.574880 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/831449be-effc-4efe-8005-f53a501168a1-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.583035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.582863 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76sw2\" (UniqueName: \"kubernetes.io/projected/831449be-effc-4efe-8005-f53a501168a1-kube-api-access-76sw2\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.584223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.584167 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:29.613993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.613966 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsnfx\" (UniqueName: \"kubernetes.io/projected/9b7b703f-6c31-4881-8e71-a23b27c3e6e7-kube-api-access-xsnfx\") pod \"payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88\" (UID: \"9b7b703f-6c31-4881-8e71-a23b27c3e6e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88" Nov 03 02:02:29.614122 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.614039 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7b703f-6c31-4881-8e71-a23b27c3e6e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88\" (UID: \"9b7b703f-6c31-4881-8e71-a23b27c3e6e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88" Nov 03 02:02:29.614824 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.614800 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7b703f-6c31-4881-8e71-a23b27c3e6e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88\" (UID: \"9b7b703f-6c31-4881-8e71-a23b27c3e6e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88" Nov 03 02:02:29.644201 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.644167 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:29.679551 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.679529 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsnfx\" (UniqueName: \"kubernetes.io/projected/9b7b703f-6c31-4881-8e71-a23b27c3e6e7-kube-api-access-xsnfx\") pod \"payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88\" (UID: \"9b7b703f-6c31-4881-8e71-a23b27c3e6e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88" Nov 03 02:02:29.726086 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.726025 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88" Nov 03 02:02:29.974993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.974946 5463 patch_prober.go:29] interesting pod/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 03 02:02:29.975169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:29.975021 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" podUID="9c143c15-d9ea-4a97-81ed-04fa713fc5d7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:02:30.057523 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:02:30.057488 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod831449be_effc_4efe_8005_f53a501168a1.slice/crio-5a87672cca5cee2f2d0608232454a788a7ff680c4269a0a1a8f124878cc20851 WatchSource:0}: Error finding container 5a87672cca5cee2f2d0608232454a788a7ff680c4269a0a1a8f124878cc20851: Status 404 returned error can't find the container with id 5a87672cca5cee2f2d0608232454a788a7ff680c4269a0a1a8f124878cc20851 Nov 03 02:02:30.060065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:30.060043 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24"] Nov 03 02:02:30.060599 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:30.060558 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:30.060675 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:30.060656 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:30.060722 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:30.060706 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:30.105733 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:30.105711 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88"] Nov 03 02:02:30.116908 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:02:30.116878 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7b703f_6c31_4881_8e71_a23b27c3e6e7.slice/crio-d582089b475d3ff3797a262fc9b59660ba5fec49bcf0458b733bc62269bef8b2 WatchSource:0}: Error finding container d582089b475d3ff3797a262fc9b59660ba5fec49bcf0458b733bc62269bef8b2: Status 404 returned error can't find the container with id d582089b475d3ff3797a262fc9b59660ba5fec49bcf0458b733bc62269bef8b2 Nov 03 02:02:30.220610 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:30.220574 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88" event={"ID":"9b7b703f-6c31-4881-8e71-a23b27c3e6e7","Type":"ContainerStarted","Data":"cdef056da06131eb9232cb318b2df99b2afce31099086cb00998882adae253c5"} Nov 03 02:02:30.220766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:30.220632 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88" event={"ID":"9b7b703f-6c31-4881-8e71-a23b27c3e6e7","Type":"ContainerStarted","Data":"d582089b475d3ff3797a262fc9b59660ba5fec49bcf0458b733bc62269bef8b2"} Nov 03 02:02:30.221310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:30.221289 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88" Nov 03 02:02:30.235747 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:30.235707 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" event={"ID":"831449be-effc-4efe-8005-f53a501168a1","Type":"ContainerStarted","Data":"4533b5687e519f60e97c2829a3c3aa50ae280db490914a70217ea31ef604248b"} Nov 03 02:02:30.235870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:30.235797 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" event={"ID":"831449be-effc-4efe-8005-f53a501168a1","Type":"ContainerStarted","Data":"5a87672cca5cee2f2d0608232454a788a7ff680c4269a0a1a8f124878cc20851"} Nov 03 02:02:30.336887 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:30.336782 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88" podStartSLOduration=1.336763109 podStartE2EDuration="1.336763109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:30.333707285 +0000 UTC m=+632.017816439" watchObservedRunningTime="2026-11-03 02:02:30.336763109 +0000 UTC m=+632.020872265" Nov 03 02:02:30.441831 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:30.441800 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86377347-5a8a-4be2-ba1d-a333b78c57ec" path="/var/lib/kubelet/pods/86377347-5a8a-4be2-ba1d-a333b78c57ec/volumes" Nov 03 02:02:30.442236 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:30.442180 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0a0798f-c015-4dc1-b468-c764722cb2ff" path="/var/lib/kubelet/pods/b0a0798f-c015-4dc1-b468-c764722cb2ff/volumes" Nov 03 02:02:30.452211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:30.452154 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" podStartSLOduration=1.452140985 podStartE2EDuration="1.452140985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:30.449454311 +0000 UTC m=+632.133563467" watchObservedRunningTime="2026-11-03 02:02:30.452140985 +0000 UTC m=+632.136250139" Nov 03 02:02:30.453060 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:30.453021 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:30.596667 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:30.596576 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:30.644590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:30.644561 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:31.650569 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:31.650541 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:32.248036 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:32.248001 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:32.248991 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:32.248973 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:02:32.602959 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:32.602879 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:32.767783 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:32.767748 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:32.913791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:32.913763 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:33.239677 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.239648 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:02:33.252922 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.252900 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-workload-certs\") pod \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " Nov 03 02:02:33.253052 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.252976 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-token\" (UniqueName: \"kubernetes.io/projected/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-token\") pod \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " Nov 03 02:02:33.253052 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.253009 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-credential-socket\") pod \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " Nov 03 02:02:33.253052 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.253049 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-workload-socket\") pod \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " Nov 03 02:02:33.253219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.253066 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-kube-api-access-wh824\" (UniqueName: \"kubernetes.io/projected/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-kube-api-access-wh824\") pod \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " Nov 03 02:02:33.253219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.253097 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-podinfo\") pod \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " Nov 03 02:02:33.253219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.253130 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-data\") pod \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " Nov 03 02:02:33.253219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.253165 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-workload-certs" pod "9c143c15-d9ea-4a97-81ed-04fa713fc5d7" (UID: "9c143c15-d9ea-4a97-81ed-04fa713fc5d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:33.253418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.253363 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-workload-socket" pod "9c143c15-d9ea-4a97-81ed-04fa713fc5d7" (UID: "9c143c15-d9ea-4a97-81ed-04fa713fc5d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:33.253484 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.253390 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-credential-socket" pod "9c143c15-d9ea-4a97-81ed-04fa713fc5d7" (UID: "9c143c15-d9ea-4a97-81ed-04fa713fc5d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:33.253668 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.253620 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-data" pod "9c143c15-d9ea-4a97-81ed-04fa713fc5d7" (UID: "9c143c15-d9ea-4a97-81ed-04fa713fc5d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:33.253808 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.253783 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-envoy\") pod \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " Nov 03 02:02:33.253888 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.253853 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istiod-ca-cert\") pod \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\" (UID: \"9c143c15-d9ea-4a97-81ed-04fa713fc5d7\") " Nov 03 02:02:33.254302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.254282 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istiod-ca-cert" pod "9c143c15-d9ea-4a97-81ed-04fa713fc5d7" (UID: "9c143c15-d9ea-4a97-81ed-04fa713fc5d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:02:33.254409 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.254393 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:33.254453 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.254415 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:33.254453 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.254429 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:33.254453 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.254445 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:33.254547 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.254460 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:33.255250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.255223 5463 generic.go:356] "Generic (PLEG): container finished" podID="9c143c15-d9ea-4a97-81ed-04fa713fc5d7" containerID="21f72345861d4eb204999c98f479b7fd0eaa645e91a31b9e219507253d32cd9e" exitCode=0 Nov 03 02:02:33.255419 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.255403 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" Nov 03 02:02:33.255657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.255631 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-token" pod "9c143c15-d9ea-4a97-81ed-04fa713fc5d7" (UID: "9c143c15-d9ea-4a97-81ed-04fa713fc5d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:33.255657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.255640 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" event={"ID":"9c143c15-d9ea-4a97-81ed-04fa713fc5d7","Type":"ContainerDied","Data":"21f72345861d4eb204999c98f479b7fd0eaa645e91a31b9e219507253d32cd9e"} Nov 03 02:02:33.255838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.255643 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-envoy" pod "9c143c15-d9ea-4a97-81ed-04fa713fc5d7" (UID: "9c143c15-d9ea-4a97-81ed-04fa713fc5d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:33.255838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.255713 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj" event={"ID":"9c143c15-d9ea-4a97-81ed-04fa713fc5d7","Type":"ContainerDied","Data":"79270060222d81bc5841112cd4a86c03fd1cd1544620c99ad2b69aa9ccc95e8c"} Nov 03 02:02:33.255838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.255742 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-podinfo" pod "9c143c15-d9ea-4a97-81ed-04fa713fc5d7" (UID: "9c143c15-d9ea-4a97-81ed-04fa713fc5d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:02:33.255838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.255759 5463 scope.go:122] "RemoveContainer" containerID="21f72345861d4eb204999c98f479b7fd0eaa645e91a31b9e219507253d32cd9e" Nov 03 02:02:33.256169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.256150 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-kube-api-access-wh824" pod "9c143c15-d9ea-4a97-81ed-04fa713fc5d7" (UID: "9c143c15-d9ea-4a97-81ed-04fa713fc5d7"). InnerVolumeSpecName "kube-api-access-wh824". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:33.275073 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.275051 5463 scope.go:122] "RemoveContainer" containerID="c9a8969a095d1893a0c94c5e669a090cf3dad6685712b9a6f020b328e6187d1a" Nov 03 02:02:33.283283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.283267 5463 scope.go:122] "RemoveContainer" containerID="21f72345861d4eb204999c98f479b7fd0eaa645e91a31b9e219507253d32cd9e" Nov 03 02:02:33.283531 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:02:33.283511 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21f72345861d4eb204999c98f479b7fd0eaa645e91a31b9e219507253d32cd9e\": container with ID starting with 21f72345861d4eb204999c98f479b7fd0eaa645e91a31b9e219507253d32cd9e not found: ID does not exist" containerID="21f72345861d4eb204999c98f479b7fd0eaa645e91a31b9e219507253d32cd9e" Nov 03 02:02:33.283579 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.283540 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21f72345861d4eb204999c98f479b7fd0eaa645e91a31b9e219507253d32cd9e"} err="failed to get container status \"21f72345861d4eb204999c98f479b7fd0eaa645e91a31b9e219507253d32cd9e\": rpc error: code = NotFound desc = could not find container \"21f72345861d4eb204999c98f479b7fd0eaa645e91a31b9e219507253d32cd9e\": container with ID starting with 21f72345861d4eb204999c98f479b7fd0eaa645e91a31b9e219507253d32cd9e not found: ID does not exist" Nov 03 02:02:33.283579 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.283558 5463 scope.go:122] "RemoveContainer" containerID="c9a8969a095d1893a0c94c5e669a090cf3dad6685712b9a6f020b328e6187d1a" Nov 03 02:02:33.283829 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:02:33.283811 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a8969a095d1893a0c94c5e669a090cf3dad6685712b9a6f020b328e6187d1a\": container with ID starting with c9a8969a095d1893a0c94c5e669a090cf3dad6685712b9a6f020b328e6187d1a not found: ID does not exist" containerID="c9a8969a095d1893a0c94c5e669a090cf3dad6685712b9a6f020b328e6187d1a" Nov 03 02:02:33.283870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.283839 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a8969a095d1893a0c94c5e669a090cf3dad6685712b9a6f020b328e6187d1a"} err="failed to get container status \"c9a8969a095d1893a0c94c5e669a090cf3dad6685712b9a6f020b328e6187d1a\": rpc error: code = NotFound desc = could not find container \"c9a8969a095d1893a0c94c5e669a090cf3dad6685712b9a6f020b328e6187d1a\": container with ID starting with c9a8969a095d1893a0c94c5e669a090cf3dad6685712b9a6f020b328e6187d1a not found: ID does not exist" Nov 03 02:02:33.356218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.356148 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:33.356218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.356180 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh824\" (UniqueName: \"kubernetes.io/projected/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-kube-api-access-wh824\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:33.356218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.356215 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:33.356645 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.356229 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-d9ea-4a97-81ed-04fa713fc5d7-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:33.407966 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.407928 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:33.571600 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.571575 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:33.678606 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.678572 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj"] Nov 03 02:02:33.681500 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:33.681477 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-fbf24ed8-openshift-default-78799b7f65-9pmhj"] Nov 03 02:02:34.165828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:34.165804 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c" Nov 03 02:02:34.166228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:34.165903 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn" Nov 03 02:02:34.444014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:34.443946 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c143c15-d9ea-4a97-81ed-04fa713fc5d7" path="/var/lib/kubelet/pods/9c143c15-d9ea-4a97-81ed-04fa713fc5d7/volumes" Nov 03 02:02:37.313808 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:37.313773 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:38.567177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:38.567144 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:38.743118 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:38.743090 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:38.966153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:38.966116 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68"] Nov 03 02:02:38.966620 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:38.966591 5463 cpu_manager.go:414] "containerMap: removing container" podUID="9c143c15-d9ea-4a97-81ed-04fa713fc5d7" containerName="istio-proxy" Nov 03 02:02:38.966620 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:38.966606 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c143c15-d9ea-4a97-81ed-04fa713fc5d7" containerName="istio-proxy" Nov 03 02:02:38.966709 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:38.966701 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c143c15-d9ea-4a97-81ed-04fa713fc5d7" containerName="istio-proxy" Nov 03 02:02:38.969926 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:38.969907 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68" Nov 03 02:02:39.015139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.014952 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-677c6\" (UniqueName: \"kubernetes.io/projected/fcd015dd-30d1-4602-9c58-e7f12e5193d7-kube-api-access-677c6\") pod \"payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68\" (UID: \"fcd015dd-30d1-4602-9c58-e7f12e5193d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68" Nov 03 02:02:39.015139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.015022 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd015dd-30d1-4602-9c58-e7f12e5193d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68\" (UID: \"fcd015dd-30d1-4602-9c58-e7f12e5193d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68" Nov 03 02:02:39.023363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.023333 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68"] Nov 03 02:02:39.046641 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.046616 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-1bccbd\"" Nov 03 02:02:39.069515 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.069491 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-1bccbd-dockercfg-4rdhd\"" Nov 03 02:02:39.073958 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.073932 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:39.116987 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.115794 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-677c6\" (UniqueName: \"kubernetes.io/projected/fcd015dd-30d1-4602-9c58-e7f12e5193d7-kube-api-access-677c6\") pod \"payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68\" (UID: \"fcd015dd-30d1-4602-9c58-e7f12e5193d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68" Nov 03 02:02:39.116987 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.115871 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd015dd-30d1-4602-9c58-e7f12e5193d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68\" (UID: \"fcd015dd-30d1-4602-9c58-e7f12e5193d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68" Nov 03 02:02:39.116987 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.116930 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd015dd-30d1-4602-9c58-e7f12e5193d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68\" (UID: \"fcd015dd-30d1-4602-9c58-e7f12e5193d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68" Nov 03 02:02:39.117955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.117926 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl"] Nov 03 02:02:39.119217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.119163 5463 cpu_manager.go:414] "containerMap: removing container" podUID="9c143c15-d9ea-4a97-81ed-04fa713fc5d7" containerName="istio-proxy" Nov 03 02:02:39.119217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.119211 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c143c15-d9ea-4a97-81ed-04fa713fc5d7" containerName="istio-proxy" Nov 03 02:02:39.119402 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.119349 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c143c15-d9ea-4a97-81ed-04fa713fc5d7" containerName="istio-proxy" Nov 03 02:02:39.122281 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.122171 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl" Nov 03 02:02:39.172411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.172380 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl"] Nov 03 02:02:39.174149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.174116 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-677c6\" (UniqueName: \"kubernetes.io/projected/fcd015dd-30d1-4602-9c58-e7f12e5193d7-kube-api-access-677c6\") pod \"payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68\" (UID: \"fcd015dd-30d1-4602-9c58-e7f12e5193d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68" Nov 03 02:02:39.217369 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.217296 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160c3088-0ac6-4694-80b9-67537f5be9eb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl\" (UID: \"160c3088-0ac6-4694-80b9-67537f5be9eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl" Nov 03 02:02:39.217369 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.217366 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnjtv\" (UniqueName: \"kubernetes.io/projected/160c3088-0ac6-4694-80b9-67537f5be9eb-kube-api-access-fnjtv\") pod \"payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl\" (UID: \"160c3088-0ac6-4694-80b9-67537f5be9eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl" Nov 03 02:02:39.239616 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.239592 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:39.282644 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.282611 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68" Nov 03 02:02:39.319127 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.319041 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160c3088-0ac6-4694-80b9-67537f5be9eb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl\" (UID: \"160c3088-0ac6-4694-80b9-67537f5be9eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl" Nov 03 02:02:39.319292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.319269 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnjtv\" (UniqueName: \"kubernetes.io/projected/160c3088-0ac6-4694-80b9-67537f5be9eb-kube-api-access-fnjtv\") pod \"payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl\" (UID: \"160c3088-0ac6-4694-80b9-67537f5be9eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl" Nov 03 02:02:39.320067 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.320035 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160c3088-0ac6-4694-80b9-67537f5be9eb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl\" (UID: \"160c3088-0ac6-4694-80b9-67537f5be9eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl" Nov 03 02:02:39.382135 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.382070 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnjtv\" (UniqueName: \"kubernetes.io/projected/160c3088-0ac6-4694-80b9-67537f5be9eb-kube-api-access-fnjtv\") pod \"payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl\" (UID: \"160c3088-0ac6-4694-80b9-67537f5be9eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl" Nov 03 02:02:39.387663 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.387636 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:39.464052 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.464027 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl" Nov 03 02:02:39.653254 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:02:39.653228 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd015dd_30d1_4602_9c58_e7f12e5193d7.slice/crio-8d8e3927e50514cb999eb643958b0b5ec2a95ff1a45a07ac4b937818138e7de5 WatchSource:0}: Error finding container 8d8e3927e50514cb999eb643958b0b5ec2a95ff1a45a07ac4b937818138e7de5: Status 404 returned error can't find the container with id 8d8e3927e50514cb999eb643958b0b5ec2a95ff1a45a07ac4b937818138e7de5 Nov 03 02:02:39.656224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.656183 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68"] Nov 03 02:02:39.835002 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:02:39.834970 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160c3088_0ac6_4694_80b9_67537f5be9eb.slice/crio-ebe58db55413513ce7b2c90e2e8c8471bc373111909df54d8dd28b88336c821b WatchSource:0}: Error finding container ebe58db55413513ce7b2c90e2e8c8471bc373111909df54d8dd28b88336c821b: Status 404 returned error can't find the container with id ebe58db55413513ce7b2c90e2e8c8471bc373111909df54d8dd28b88336c821b Nov 03 02:02:39.837394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.837370 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl"] Nov 03 02:02:39.935972 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:39.935939 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:40.162244 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:40.162214 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:40.300753 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:40.300707 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl" event={"ID":"160c3088-0ac6-4694-80b9-67537f5be9eb","Type":"ContainerStarted","Data":"6b3e81d8476dbe1fcbead8e795568c71a87f79b7f82ecbe71688f894a306359d"} Nov 03 02:02:40.300936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:40.300796 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl" event={"ID":"160c3088-0ac6-4694-80b9-67537f5be9eb","Type":"ContainerStarted","Data":"ebe58db55413513ce7b2c90e2e8c8471bc373111909df54d8dd28b88336c821b"} Nov 03 02:02:40.300936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:40.300852 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl" Nov 03 02:02:40.302363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:40.302334 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68" event={"ID":"fcd015dd-30d1-4602-9c58-e7f12e5193d7","Type":"ContainerStarted","Data":"db6ae0eabc862a6d2b0f2d7daad8ab05880932a1eed1f29eeedfe32ca8f7316f"} Nov 03 02:02:40.302480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:40.302383 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68" event={"ID":"fcd015dd-30d1-4602-9c58-e7f12e5193d7","Type":"ContainerStarted","Data":"8d8e3927e50514cb999eb643958b0b5ec2a95ff1a45a07ac4b937818138e7de5"} Nov 03 02:02:40.302480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:40.302424 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68" Nov 03 02:02:40.418568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:40.418474 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl" podStartSLOduration=1.418460443 podStartE2EDuration="1.418460443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:40.416658859 +0000 UTC m=+642.100768015" watchObservedRunningTime="2026-11-03 02:02:40.418460443 +0000 UTC m=+642.102569595" Nov 03 02:02:40.533743 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:40.533694 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68" podStartSLOduration=2.53367829 podStartE2EDuration="2.53367829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:40.531535957 +0000 UTC m=+642.215645117" watchObservedRunningTime="2026-11-03 02:02:40.53367829 +0000 UTC m=+642.217787447" Nov 03 02:02:41.918783 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:41.918745 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz"] Nov 03 02:02:41.919259 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:41.918991 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" podUID="cc06054f-00b6-4308-ac4b-a315a36ef0ba" containerName="istio-proxy" containerID="cri-o://81a2907b67c30e18ee93bb06f607d9787c4601f915562bf6cc77791958edf854" gracePeriod=30 Nov 03 02:02:41.936493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:41.936469 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:42.251296 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:42.251215 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88" Nov 03 02:02:42.798112 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:42.798074 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:43.006481 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:02:43.006450 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:02:43.281381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.281346 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p"] Nov 03 02:02:43.284340 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.284320 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.347709 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.347659 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c7d758-openshift-default-dockercfg-kst7d\"" Nov 03 02:02:43.374698 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.374665 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.374850 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.374723 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.374850 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.374833 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2746f922-b469-4383-90bb-3be08f9256d7-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.374982 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.374885 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2746f922-b469-4383-90bb-3be08f9256d7-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.374982 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.374910 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2746f922-b469-4383-90bb-3be08f9256d7-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.375129 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.375065 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7vz9\" (UniqueName: \"kubernetes.io/projected/2746f922-b469-4383-90bb-3be08f9256d7-kube-api-access-w7vz9\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.375129 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.375090 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.375129 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.375120 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.375319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.375293 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.394341 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.394309 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p"] Nov 03 02:02:43.406474 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.406440 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:43.476524 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.476493 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.476693 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.476552 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.476783 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.476764 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.476844 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.476805 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2746f922-b469-4383-90bb-3be08f9256d7-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.476909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.476845 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2746f922-b469-4383-90bb-3be08f9256d7-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.476909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.476896 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2746f922-b469-4383-90bb-3be08f9256d7-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.477008 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.476908 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.477008 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.476964 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.477100 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.477031 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7vz9\" (UniqueName: \"kubernetes.io/projected/2746f922-b469-4383-90bb-3be08f9256d7-kube-api-access-w7vz9\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.477100 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.477069 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.477226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.477116 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.477226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.477119 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.477542 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.477518 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.477665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.477641 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2746f922-b469-4383-90bb-3be08f9256d7-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.479485 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.479461 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2746f922-b469-4383-90bb-3be08f9256d7-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.479657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.479640 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.534577 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.534473 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2746f922-b469-4383-90bb-3be08f9256d7-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.534720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.534639 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7vz9\" (UniqueName: \"kubernetes.io/projected/2746f922-b469-4383-90bb-3be08f9256d7-kube-api-access-w7vz9\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:43.565780 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.565752 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:43.596917 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:43.596889 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:44.007102 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:02:44.007073 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2746f922_b469_4383_90bb_3be08f9256d7.slice/crio-a5fff26fd56eba8e04ad5a9c5193cbf9ac045c2928ae6acc00fee7ddf457ef37 WatchSource:0}: Error finding container a5fff26fd56eba8e04ad5a9c5193cbf9ac045c2928ae6acc00fee7ddf457ef37: Status 404 returned error can't find the container with id a5fff26fd56eba8e04ad5a9c5193cbf9ac045c2928ae6acc00fee7ddf457ef37 Nov 03 02:02:44.009643 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:44.009603 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:44.009720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:44.009688 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:44.009720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:44.009701 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p"] Nov 03 02:02:44.009796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:44.009729 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:44.325353 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:44.325267 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:44.342145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:44.342108 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" event={"ID":"2746f922-b469-4383-90bb-3be08f9256d7","Type":"ContainerStarted","Data":"4665c8b21421aec75f704c6ad2b2457e3cf9c6bfd07b05cf5a00877084b195e2"} Nov 03 02:02:44.342344 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:44.342162 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" event={"ID":"2746f922-b469-4383-90bb-3be08f9256d7","Type":"ContainerStarted","Data":"a5fff26fd56eba8e04ad5a9c5193cbf9ac045c2928ae6acc00fee7ddf457ef37"} Nov 03 02:02:44.458117 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:44.458062 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" podStartSLOduration=1.458047921 podStartE2EDuration="1.458047921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:44.457100882 +0000 UTC m=+646.141210036" watchObservedRunningTime="2026-11-03 02:02:44.458047921 +0000 UTC m=+646.142157076" Nov 03 02:02:44.597977 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:44.597904 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:45.602247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:45.602220 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:46.353610 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:46.353575 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:46.354497 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:46.354477 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:02:47.219972 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.219951 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:02:47.331626 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.331593 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:47.333118 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.333092 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istiod-ca-cert\") pod \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " Nov 03 02:02:47.333252 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.333157 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-credential-socket\") pod \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " Nov 03 02:02:47.333252 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.333221 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-workload-certs\") pod \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " Nov 03 02:02:47.333358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.333254 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-envoy\") pod \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " Nov 03 02:02:47.333358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.333318 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc06054f-00b6-4308-ac4b-a315a36ef0ba-kube-api-access-qf8sr\" (UniqueName: \"kubernetes.io/projected/cc06054f-00b6-4308-ac4b-a315a36ef0ba-kube-api-access-qf8sr\") pod \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " Nov 03 02:02:47.333456 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.333354 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-token\" (UniqueName: \"kubernetes.io/projected/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-token\") pod \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " Nov 03 02:02:47.333456 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.333390 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-workload-socket\") pod \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " Nov 03 02:02:47.333456 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.333434 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-podinfo\") pod \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " Nov 03 02:02:47.333609 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.333464 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-credential-socket" pod "cc06054f-00b6-4308-ac4b-a315a36ef0ba" (UID: "cc06054f-00b6-4308-ac4b-a315a36ef0ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:47.333609 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.333487 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-data\") pod \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\" (UID: \"cc06054f-00b6-4308-ac4b-a315a36ef0ba\") " Nov 03 02:02:47.333714 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.333614 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-workload-certs" pod "cc06054f-00b6-4308-ac4b-a315a36ef0ba" (UID: "cc06054f-00b6-4308-ac4b-a315a36ef0ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:47.333714 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.333637 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istiod-ca-cert" pod "cc06054f-00b6-4308-ac4b-a315a36ef0ba" (UID: "cc06054f-00b6-4308-ac4b-a315a36ef0ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:02:47.333819 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.333801 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-workload-socket" pod "cc06054f-00b6-4308-ac4b-a315a36ef0ba" (UID: "cc06054f-00b6-4308-ac4b-a315a36ef0ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:47.333897 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.333877 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-data" pod "cc06054f-00b6-4308-ac4b-a315a36ef0ba" (UID: "cc06054f-00b6-4308-ac4b-a315a36ef0ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:47.333962 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.333942 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:47.334057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.333966 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:47.334057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.333981 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:47.334057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.333996 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:47.334057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.334015 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:47.335653 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.335628 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-podinfo" pod "cc06054f-00b6-4308-ac4b-a315a36ef0ba" (UID: "cc06054f-00b6-4308-ac4b-a315a36ef0ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:02:47.335922 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.335896 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc06054f-00b6-4308-ac4b-a315a36ef0ba-kube-api-access-qf8sr" pod "cc06054f-00b6-4308-ac4b-a315a36ef0ba" (UID: "cc06054f-00b6-4308-ac4b-a315a36ef0ba"). InnerVolumeSpecName "kube-api-access-qf8sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:47.336055 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.336037 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-envoy" pod "cc06054f-00b6-4308-ac4b-a315a36ef0ba" (UID: "cc06054f-00b6-4308-ac4b-a315a36ef0ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:47.336139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.336120 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-token" pod "cc06054f-00b6-4308-ac4b-a315a36ef0ba" (UID: "cc06054f-00b6-4308-ac4b-a315a36ef0ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:47.367444 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.367413 5463 generic.go:356] "Generic (PLEG): container finished" podID="cc06054f-00b6-4308-ac4b-a315a36ef0ba" containerID="81a2907b67c30e18ee93bb06f607d9787c4601f915562bf6cc77791958edf854" exitCode=0 Nov 03 02:02:47.367567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.367488 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" Nov 03 02:02:47.367567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.367497 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" event={"ID":"cc06054f-00b6-4308-ac4b-a315a36ef0ba","Type":"ContainerDied","Data":"81a2907b67c30e18ee93bb06f607d9787c4601f915562bf6cc77791958edf854"} Nov 03 02:02:47.367658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.367586 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz" event={"ID":"cc06054f-00b6-4308-ac4b-a315a36ef0ba","Type":"ContainerDied","Data":"9351807f7b87ece46510cf755eb6c11893ade7a947116008d0f743096ddea117"} Nov 03 02:02:47.367658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.367608 5463 scope.go:122] "RemoveContainer" containerID="81a2907b67c30e18ee93bb06f607d9787c4601f915562bf6cc77791958edf854" Nov 03 02:02:47.377362 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.377342 5463 scope.go:122] "RemoveContainer" containerID="34a8539a377ca9d6fd6acaa63603a5715158a5be41a13052018289cd55c66fe3" Nov 03 02:02:47.385434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.385418 5463 scope.go:122] "RemoveContainer" containerID="81a2907b67c30e18ee93bb06f607d9787c4601f915562bf6cc77791958edf854" Nov 03 02:02:47.385711 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:02:47.385695 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a2907b67c30e18ee93bb06f607d9787c4601f915562bf6cc77791958edf854\": container with ID starting with 81a2907b67c30e18ee93bb06f607d9787c4601f915562bf6cc77791958edf854 not found: ID does not exist" containerID="81a2907b67c30e18ee93bb06f607d9787c4601f915562bf6cc77791958edf854" Nov 03 02:02:47.385755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.385716 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a2907b67c30e18ee93bb06f607d9787c4601f915562bf6cc77791958edf854"} err="failed to get container status \"81a2907b67c30e18ee93bb06f607d9787c4601f915562bf6cc77791958edf854\": rpc error: code = NotFound desc = could not find container \"81a2907b67c30e18ee93bb06f607d9787c4601f915562bf6cc77791958edf854\": container with ID starting with 81a2907b67c30e18ee93bb06f607d9787c4601f915562bf6cc77791958edf854 not found: ID does not exist" Nov 03 02:02:47.385755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.385731 5463 scope.go:122] "RemoveContainer" containerID="34a8539a377ca9d6fd6acaa63603a5715158a5be41a13052018289cd55c66fe3" Nov 03 02:02:47.385969 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:02:47.385950 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34a8539a377ca9d6fd6acaa63603a5715158a5be41a13052018289cd55c66fe3\": container with ID starting with 34a8539a377ca9d6fd6acaa63603a5715158a5be41a13052018289cd55c66fe3 not found: ID does not exist" containerID="34a8539a377ca9d6fd6acaa63603a5715158a5be41a13052018289cd55c66fe3" Nov 03 02:02:47.386019 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.385975 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34a8539a377ca9d6fd6acaa63603a5715158a5be41a13052018289cd55c66fe3"} err="failed to get container status \"34a8539a377ca9d6fd6acaa63603a5715158a5be41a13052018289cd55c66fe3\": rpc error: code = NotFound desc = could not find container \"34a8539a377ca9d6fd6acaa63603a5715158a5be41a13052018289cd55c66fe3\": container with ID starting with 34a8539a377ca9d6fd6acaa63603a5715158a5be41a13052018289cd55c66fe3 not found: ID does not exist" Nov 03 02:02:47.435001 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.434968 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:47.435001 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.434998 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:47.435268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.435015 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf8sr\" (UniqueName: \"kubernetes.io/projected/cc06054f-00b6-4308-ac4b-a315a36ef0ba-kube-api-access-qf8sr\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:47.435268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.435029 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc06054f-00b6-4308-ac4b-a315a36ef0ba-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:47.483900 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.483806 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:47.507005 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.506963 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz"] Nov 03 02:02:47.510095 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.510070 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-aead6b17-openshift-default-65fc747d9f-qlwnz"] Nov 03 02:02:47.636821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:47.636790 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:48.442670 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:48.442638 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc06054f-00b6-4308-ac4b-a315a36ef0ba" path="/var/lib/kubelet/pods/cc06054f-00b6-4308-ac4b-a315a36ef0ba/volumes" Nov 03 02:02:48.738486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:48.738347 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:50.522439 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:50.522408 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:50.673751 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:50.673718 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:50.820739 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:50.820665 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:50.898662 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:50.898632 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7"] Nov 03 02:02:50.899204 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:50.899175 5463 cpu_manager.go:414] "containerMap: removing container" podUID="cc06054f-00b6-4308-ac4b-a315a36ef0ba" containerName="istio-proxy" Nov 03 02:02:50.899356 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:50.899209 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc06054f-00b6-4308-ac4b-a315a36ef0ba" containerName="istio-proxy" Nov 03 02:02:50.899356 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:50.899239 5463 cpu_manager.go:414] "containerMap: removing container" podUID="cc06054f-00b6-4308-ac4b-a315a36ef0ba" containerName="istio-proxy" Nov 03 02:02:50.899356 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:50.899245 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc06054f-00b6-4308-ac4b-a315a36ef0ba" containerName="istio-proxy" Nov 03 02:02:50.899550 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:50.899390 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc06054f-00b6-4308-ac4b-a315a36ef0ba" containerName="istio-proxy" Nov 03 02:02:50.899550 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:50.899408 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc06054f-00b6-4308-ac4b-a315a36ef0ba" containerName="istio-proxy" Nov 03 02:02:50.902509 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:50.902491 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7" Nov 03 02:02:50.967216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:50.963296 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7"] Nov 03 02:02:50.994223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:50.990157 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32df4d57-7e5b-4136-9e75-5c4e9290fd30-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7\" (UID: \"32df4d57-7e5b-4136-9e75-5c4e9290fd30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7" Nov 03 02:02:50.994223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:50.990260 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7c9p\" (UniqueName: \"kubernetes.io/projected/32df4d57-7e5b-4136-9e75-5c4e9290fd30-kube-api-access-m7c9p\") pod \"payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7\" (UID: \"32df4d57-7e5b-4136-9e75-5c4e9290fd30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7" Nov 03 02:02:51.012230 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:51.012168 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-c7d758-dockercfg-xp9kq\"" Nov 03 02:02:51.012451 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:51.012241 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-c7d758\"" Nov 03 02:02:51.091482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:51.091395 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32df4d57-7e5b-4136-9e75-5c4e9290fd30-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7\" (UID: \"32df4d57-7e5b-4136-9e75-5c4e9290fd30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7" Nov 03 02:02:51.091482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:51.091437 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7c9p\" (UniqueName: \"kubernetes.io/projected/32df4d57-7e5b-4136-9e75-5c4e9290fd30-kube-api-access-m7c9p\") pod \"payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7\" (UID: \"32df4d57-7e5b-4136-9e75-5c4e9290fd30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7" Nov 03 02:02:51.092674 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:51.092646 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32df4d57-7e5b-4136-9e75-5c4e9290fd30-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7\" (UID: \"32df4d57-7e5b-4136-9e75-5c4e9290fd30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7" Nov 03 02:02:51.176576 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:51.175922 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7c9p\" (UniqueName: \"kubernetes.io/projected/32df4d57-7e5b-4136-9e75-5c4e9290fd30-kube-api-access-m7c9p\") pod \"payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7\" (UID: \"32df4d57-7e5b-4136-9e75-5c4e9290fd30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7" Nov 03 02:02:51.215027 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:51.215004 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7" Nov 03 02:02:51.312482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:51.312454 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68" Nov 03 02:02:51.312698 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:51.312677 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl" Nov 03 02:02:51.678016 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:51.677980 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7"] Nov 03 02:02:51.773214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:51.770407 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:51.963273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:51.962447 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:52.174836 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:52.174768 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r"] Nov 03 02:02:52.176636 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:52.176614 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-080015ba-6vj4r"] Nov 03 02:02:52.419181 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:52.419146 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7" event={"ID":"32df4d57-7e5b-4136-9e75-5c4e9290fd30","Type":"ContainerStarted","Data":"09860774bae4e8bca0e5fed3180b95a45c4fea28047e0801612cd93c1ff1e7e9"} Nov 03 02:02:52.419362 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:52.419218 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7" event={"ID":"32df4d57-7e5b-4136-9e75-5c4e9290fd30","Type":"ContainerStarted","Data":"f2e9f4bc29f0725ff844b6075ff4982542c31e16aba0984ed5ef4c7f3eefef67"} Nov 03 02:02:52.419947 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:52.419926 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7" Nov 03 02:02:52.441605 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:52.441580 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0af47c99-d045-406c-8e5e-c1e3662f5058" path="/var/lib/kubelet/pods/0af47c99-d045-406c-8e5e-c1e3662f5058/volumes" Nov 03 02:02:52.536155 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:52.536100 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7" podStartSLOduration=2.536086836 podStartE2EDuration="2.536086836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:52.535186009 +0000 UTC m=+654.219295163" watchObservedRunningTime="2026-11-03 02:02:52.536086836 +0000 UTC m=+654.220196032" Nov 03 02:02:52.876405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:52.876317 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw"] Nov 03 02:02:52.876835 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:52.876557 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" podUID="0a70d009-4bbc-4a9e-8545-e47d9eb399b2" containerName="istio-proxy" containerID="cri-o://64d85d1d0cb8ef879c275fe5766e0110602db66dfb897f74c375806d7c390270" gracePeriod=30 Nov 03 02:02:52.900764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:52.900722 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:53.608473 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:53.608431 5463 patch_prober.go:29] interesting pod/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:02:53.608643 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:53.608483 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" podUID="0a70d009-4bbc-4a9e-8545-e47d9eb399b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:02:53.671748 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:53.671676 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:54.181630 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.181603 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6"] Nov 03 02:02:54.185971 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.185945 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.246033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.245985 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6"] Nov 03 02:02:54.288979 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.288901 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:54.330658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.330286 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e094c575-bfcd-41a1-902a-a348adde1c14-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.330658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.330325 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.330658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.330360 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e094c575-bfcd-41a1-902a-a348adde1c14-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.330658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.330409 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.330658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.330425 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.330658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.330439 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.330658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.330472 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e094c575-bfcd-41a1-902a-a348adde1c14-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.330658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.330495 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.330658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.330529 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzmnk\" (UniqueName: \"kubernetes.io/projected/e094c575-bfcd-41a1-902a-a348adde1c14-kube-api-access-gzmnk\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.335303 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.335256 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f7d9bb-openshift-default-dockercfg-dp5lm\"" Nov 03 02:02:54.431050 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.431014 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e094c575-bfcd-41a1-902a-a348adde1c14-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.431266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.431084 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.431266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.431104 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.431266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.431119 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.431266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.431149 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e094c575-bfcd-41a1-902a-a348adde1c14-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.431266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.431175 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.431266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.431233 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzmnk\" (UniqueName: \"kubernetes.io/projected/e094c575-bfcd-41a1-902a-a348adde1c14-kube-api-access-gzmnk\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.431266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.431267 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e094c575-bfcd-41a1-902a-a348adde1c14-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.431623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.431284 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.431623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.431605 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.431763 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.431687 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.432392 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.432098 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e094c575-bfcd-41a1-902a-a348adde1c14-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.432392 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.432237 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.432392 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.432321 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.434388 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.434365 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.434535 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.434450 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e094c575-bfcd-41a1-902a-a348adde1c14-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.489050 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.489018 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzmnk\" (UniqueName: \"kubernetes.io/projected/e094c575-bfcd-41a1-902a-a348adde1c14-kube-api-access-gzmnk\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.489695 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.489677 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e094c575-bfcd-41a1-902a-a348adde1c14-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.500766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.500742 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:54.589370 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.585584 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:54.952115 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:02:54.952085 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode094c575_bfcd_41a1_902a_a348adde1c14.slice/crio-03a55ea5b09ceb94d5b04feeeddc0addc715b706f9d3dd6177e91106acf5a641 WatchSource:0}: Error finding container 03a55ea5b09ceb94d5b04feeeddc0addc715b706f9d3dd6177e91106acf5a641: Status 404 returned error can't find the container with id 03a55ea5b09ceb94d5b04feeeddc0addc715b706f9d3dd6177e91106acf5a641 Nov 03 02:02:54.952861 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.952838 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6"] Nov 03 02:02:54.955821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.955782 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:54.955915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.955867 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:54.955955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:54.955910 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:55.439983 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:55.439947 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" event={"ID":"e094c575-bfcd-41a1-902a-a348adde1c14","Type":"ContainerStarted","Data":"628f71f8b1f9ef99eb0408f733671b4fad509b7d267d2d5d1672b3ae54f2c8c8"} Nov 03 02:02:55.440487 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:55.440459 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" event={"ID":"e094c575-bfcd-41a1-902a-a348adde1c14","Type":"ContainerStarted","Data":"03a55ea5b09ceb94d5b04feeeddc0addc715b706f9d3dd6177e91106acf5a641"} Nov 03 02:02:55.501042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:55.501007 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:55.557856 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:55.557808 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" podStartSLOduration=1.557794387 podStartE2EDuration="1.557794387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:55.556341438 +0000 UTC m=+657.240450593" watchObservedRunningTime="2026-11-03 02:02:55.557794387 +0000 UTC m=+657.241903541" Nov 03 02:02:55.928784 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:55.928752 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:56.506248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:56.506218 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:57.451130 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:57.451095 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:57.452248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:57.452226 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:02:58.173583 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.173563 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:02:58.288122 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.288051 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-token\" (UniqueName: \"kubernetes.io/projected/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-token\") pod \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " Nov 03 02:02:58.288122 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.288094 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istiod-ca-cert\") pod \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " Nov 03 02:02:58.288371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.288138 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-envoy\") pod \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " Nov 03 02:02:58.288371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.288250 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-workload-socket\") pod \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " Nov 03 02:02:58.288371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.288295 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-credential-socket\") pod \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " Nov 03 02:02:58.288371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.288319 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-kube-api-access-b4gr9\" (UniqueName: \"kubernetes.io/projected/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-kube-api-access-b4gr9\") pod \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " Nov 03 02:02:58.288589 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.288373 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-podinfo\") pod \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " Nov 03 02:02:58.288589 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.288405 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-workload-certs\") pod \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " Nov 03 02:02:58.288589 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.288443 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-data\") pod \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\" (UID: \"0a70d009-4bbc-4a9e-8545-e47d9eb399b2\") " Nov 03 02:02:58.288589 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.288518 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istiod-ca-cert" pod "0a70d009-4bbc-4a9e-8545-e47d9eb399b2" (UID: "0a70d009-4bbc-4a9e-8545-e47d9eb399b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:02:58.288589 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.288510 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-workload-socket" pod "0a70d009-4bbc-4a9e-8545-e47d9eb399b2" (UID: "0a70d009-4bbc-4a9e-8545-e47d9eb399b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:58.288589 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.288574 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-credential-socket" pod "0a70d009-4bbc-4a9e-8545-e47d9eb399b2" (UID: "0a70d009-4bbc-4a9e-8545-e47d9eb399b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:58.289405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.288963 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-workload-certs" pod "0a70d009-4bbc-4a9e-8545-e47d9eb399b2" (UID: "0a70d009-4bbc-4a9e-8545-e47d9eb399b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:58.289405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.288978 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-data" pod "0a70d009-4bbc-4a9e-8545-e47d9eb399b2" (UID: "0a70d009-4bbc-4a9e-8545-e47d9eb399b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:58.289405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.288994 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:58.289405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.289013 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:58.289405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.289027 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:58.291078 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.290871 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-envoy" pod "0a70d009-4bbc-4a9e-8545-e47d9eb399b2" (UID: "0a70d009-4bbc-4a9e-8545-e47d9eb399b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:58.291078 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.290932 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-token" pod "0a70d009-4bbc-4a9e-8545-e47d9eb399b2" (UID: "0a70d009-4bbc-4a9e-8545-e47d9eb399b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:58.291279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.291078 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-kube-api-access-b4gr9" pod "0a70d009-4bbc-4a9e-8545-e47d9eb399b2" (UID: "0a70d009-4bbc-4a9e-8545-e47d9eb399b2"). InnerVolumeSpecName "kube-api-access-b4gr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:58.291774 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.291746 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-podinfo" pod "0a70d009-4bbc-4a9e-8545-e47d9eb399b2" (UID: "0a70d009-4bbc-4a9e-8545-e47d9eb399b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:02:58.390549 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.390518 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4gr9\" (UniqueName: \"kubernetes.io/projected/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-kube-api-access-b4gr9\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:58.390549 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.390545 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:58.390549 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.390555 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:58.390771 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.390563 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:58.390771 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.390573 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:58.390771 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.390581 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a70d009-4bbc-4a9e-8545-e47d9eb399b2-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:58.401319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.401244 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:58.457494 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.457471 5463 generic.go:356] "Generic (PLEG): container finished" podID="0a70d009-4bbc-4a9e-8545-e47d9eb399b2" containerID="64d85d1d0cb8ef879c275fe5766e0110602db66dfb897f74c375806d7c390270" exitCode=0 Nov 03 02:02:58.457611 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.457512 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" Nov 03 02:02:58.457611 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.457552 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" event={"ID":"0a70d009-4bbc-4a9e-8545-e47d9eb399b2","Type":"ContainerDied","Data":"64d85d1d0cb8ef879c275fe5766e0110602db66dfb897f74c375806d7c390270"} Nov 03 02:02:58.457611 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.457601 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw" event={"ID":"0a70d009-4bbc-4a9e-8545-e47d9eb399b2","Type":"ContainerDied","Data":"305fc85d02e79766c2118101fae4c13543f1f84799784f1d2680f916185a1ac7"} Nov 03 02:02:58.457770 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.457625 5463 scope.go:122] "RemoveContainer" containerID="64d85d1d0cb8ef879c275fe5766e0110602db66dfb897f74c375806d7c390270" Nov 03 02:02:58.459317 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.459292 5463 generic.go:356] "Generic (PLEG): container finished" podID="88462923-311f-4302-9503-b1dda3e244e2" containerID="fc5953001997a0d8392b007654bb435c0fbd39fad5bdfa6f8ce6b6f723276eae" exitCode=0 Nov 03 02:02:58.459471 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.459417 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" event={"ID":"88462923-311f-4302-9503-b1dda3e244e2","Type":"ContainerDied","Data":"fc5953001997a0d8392b007654bb435c0fbd39fad5bdfa6f8ce6b6f723276eae"} Nov 03 02:02:58.459809 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.459788 5463 scope.go:122] "RemoveContainer" containerID="fc5953001997a0d8392b007654bb435c0fbd39fad5bdfa6f8ce6b6f723276eae" Nov 03 02:02:58.461072 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.461044 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:58.461153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.461112 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:58.461153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.461144 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:02:58.467511 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.467496 5463 scope.go:122] "RemoveContainer" containerID="060456b20ed83ac1d32dcd23512850dff35b64807b93966098322778e7dcc755" Nov 03 02:02:58.476494 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.476477 5463 scope.go:122] "RemoveContainer" containerID="64d85d1d0cb8ef879c275fe5766e0110602db66dfb897f74c375806d7c390270" Nov 03 02:02:58.476782 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:02:58.476760 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d85d1d0cb8ef879c275fe5766e0110602db66dfb897f74c375806d7c390270\": container with ID starting with 64d85d1d0cb8ef879c275fe5766e0110602db66dfb897f74c375806d7c390270 not found: ID does not exist" containerID="64d85d1d0cb8ef879c275fe5766e0110602db66dfb897f74c375806d7c390270" Nov 03 02:02:58.476837 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.476792 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d85d1d0cb8ef879c275fe5766e0110602db66dfb897f74c375806d7c390270"} err="failed to get container status \"64d85d1d0cb8ef879c275fe5766e0110602db66dfb897f74c375806d7c390270\": rpc error: code = NotFound desc = could not find container \"64d85d1d0cb8ef879c275fe5766e0110602db66dfb897f74c375806d7c390270\": container with ID starting with 64d85d1d0cb8ef879c275fe5766e0110602db66dfb897f74c375806d7c390270 not found: ID does not exist" Nov 03 02:02:58.476837 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.476812 5463 scope.go:122] "RemoveContainer" containerID="060456b20ed83ac1d32dcd23512850dff35b64807b93966098322778e7dcc755" Nov 03 02:02:58.477115 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:02:58.477096 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"060456b20ed83ac1d32dcd23512850dff35b64807b93966098322778e7dcc755\": container with ID starting with 060456b20ed83ac1d32dcd23512850dff35b64807b93966098322778e7dcc755 not found: ID does not exist" containerID="060456b20ed83ac1d32dcd23512850dff35b64807b93966098322778e7dcc755" Nov 03 02:02:58.477180 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.477127 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"060456b20ed83ac1d32dcd23512850dff35b64807b93966098322778e7dcc755"} err="failed to get container status \"060456b20ed83ac1d32dcd23512850dff35b64807b93966098322778e7dcc755\": rpc error: code = NotFound desc = could not find container \"060456b20ed83ac1d32dcd23512850dff35b64807b93966098322778e7dcc755\": container with ID starting with 060456b20ed83ac1d32dcd23512850dff35b64807b93966098322778e7dcc755 not found: ID does not exist" Nov 03 02:02:58.573957 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.573925 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:58.630175 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.630126 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw"] Nov 03 02:02:58.636736 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.636687 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-080015ba-openshift-default-9dd5bc799-rqczw"] Nov 03 02:02:58.820122 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:58.820026 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:59.498714 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:02:59.498678 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" event={"ID":"88462923-311f-4302-9503-b1dda3e244e2","Type":"ContainerStarted","Data":"32ab147d5ad68ccace0e681d63919a5809efd83fe3ff6968aca1976e586ae3da"} Nov 03 02:03:00.357586 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:00.352543 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:03:00.357586 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:00.352681 5463 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 03 02:03:00.357586 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:00.352735 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" podUID="88462923-311f-4302-9503-b1dda3e244e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 03 02:03:00.442151 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:00.442120 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a70d009-4bbc-4a9e-8545-e47d9eb399b2" path="/var/lib/kubelet/pods/0a70d009-4bbc-4a9e-8545-e47d9eb399b2/volumes" Nov 03 02:03:01.027782 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:01.027757 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:01.357621 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:01.357530 5463 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 03 02:03:01.357894 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:01.357858 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" podUID="88462923-311f-4302-9503-b1dda3e244e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 03 02:03:01.789222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:01.789125 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:01.938827 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:01.938792 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:02.197217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.196399 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst"] Nov 03 02:03:02.197217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.197180 5463 cpu_manager.go:414] "containerMap: removing container" podUID="0a70d009-4bbc-4a9e-8545-e47d9eb399b2" containerName="istio-proxy" Nov 03 02:03:02.197217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.197218 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a70d009-4bbc-4a9e-8545-e47d9eb399b2" containerName="istio-proxy" Nov 03 02:03:02.197837 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.197237 5463 cpu_manager.go:414] "containerMap: removing container" podUID="0a70d009-4bbc-4a9e-8545-e47d9eb399b2" containerName="istio-proxy" Nov 03 02:03:02.197837 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.197246 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a70d009-4bbc-4a9e-8545-e47d9eb399b2" containerName="istio-proxy" Nov 03 02:03:02.197837 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.197399 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a70d009-4bbc-4a9e-8545-e47d9eb399b2" containerName="istio-proxy" Nov 03 02:03:02.197837 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.197411 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a70d009-4bbc-4a9e-8545-e47d9eb399b2" containerName="istio-proxy" Nov 03 02:03:02.206317 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.206292 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst" Nov 03 02:03:02.210110 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.210087 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst"] Nov 03 02:03:02.299216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.271107 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e01f46d8-b6be-4f51-9291-a67a09771b4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst\" (UID: \"e01f46d8-b6be-4f51-9291-a67a09771b4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst" Nov 03 02:03:02.303219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.299616 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5szv7\" (UniqueName: \"kubernetes.io/projected/e01f46d8-b6be-4f51-9291-a67a09771b4e-kube-api-access-5szv7\") pod \"payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst\" (UID: \"e01f46d8-b6be-4f51-9291-a67a09771b4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst" Nov 03 02:03:02.328537 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.328501 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f7d9bb-dockercfg-knjth\"" Nov 03 02:03:02.328731 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.328613 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f7d9bb\"" Nov 03 02:03:02.328870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.328844 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc"] Nov 03 02:03:02.333077 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.333054 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc" Nov 03 02:03:02.373224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.369800 5463 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 03 02:03:02.373224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.369851 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" podUID="88462923-311f-4302-9503-b1dda3e244e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 03 02:03:02.373224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.371081 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc"] Nov 03 02:03:02.413578 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.412760 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e01f46d8-b6be-4f51-9291-a67a09771b4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst\" (UID: \"e01f46d8-b6be-4f51-9291-a67a09771b4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst" Nov 03 02:03:02.413578 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.412834 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5szv7\" (UniqueName: \"kubernetes.io/projected/e01f46d8-b6be-4f51-9291-a67a09771b4e-kube-api-access-5szv7\") pod \"payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst\" (UID: \"e01f46d8-b6be-4f51-9291-a67a09771b4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst" Nov 03 02:03:02.413578 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.412917 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcc529d1-9757-467a-bde9-c0afa4822017-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc\" (UID: \"fcc529d1-9757-467a-bde9-c0afa4822017\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc" Nov 03 02:03:02.413578 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.412989 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trlct\" (UniqueName: \"kubernetes.io/projected/fcc529d1-9757-467a-bde9-c0afa4822017-kube-api-access-trlct\") pod \"payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc\" (UID: \"fcc529d1-9757-467a-bde9-c0afa4822017\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc" Nov 03 02:03:02.413994 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.413963 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e01f46d8-b6be-4f51-9291-a67a09771b4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst\" (UID: \"e01f46d8-b6be-4f51-9291-a67a09771b4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst" Nov 03 02:03:02.507427 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.507270 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5szv7\" (UniqueName: \"kubernetes.io/projected/e01f46d8-b6be-4f51-9291-a67a09771b4e-kube-api-access-5szv7\") pod \"payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst\" (UID: \"e01f46d8-b6be-4f51-9291-a67a09771b4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst" Nov 03 02:03:02.513630 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.513603 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trlct\" (UniqueName: \"kubernetes.io/projected/fcc529d1-9757-467a-bde9-c0afa4822017-kube-api-access-trlct\") pod \"payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc\" (UID: \"fcc529d1-9757-467a-bde9-c0afa4822017\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc" Nov 03 02:03:02.513778 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.513720 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcc529d1-9757-467a-bde9-c0afa4822017-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc\" (UID: \"fcc529d1-9757-467a-bde9-c0afa4822017\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc" Nov 03 02:03:02.514376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.514356 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcc529d1-9757-467a-bde9-c0afa4822017-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc\" (UID: \"fcc529d1-9757-467a-bde9-c0afa4822017\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc" Nov 03 02:03:02.525508 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.525486 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst" Nov 03 02:03:02.587919 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.587852 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trlct\" (UniqueName: \"kubernetes.io/projected/fcc529d1-9757-467a-bde9-c0afa4822017-kube-api-access-trlct\") pod \"payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc\" (UID: \"fcc529d1-9757-467a-bde9-c0afa4822017\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc" Nov 03 02:03:02.675941 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.675628 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc" Nov 03 02:03:02.998229 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:02.990223 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst"] Nov 03 02:03:03.135649 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:03:03.135624 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc529d1_9757_467a_bde9_c0afa4822017.slice/crio-7bb7ac40385d916826dd093df6f16155191667976900d80110f679b9ced6c25c WatchSource:0}: Error finding container 7bb7ac40385d916826dd093df6f16155191667976900d80110f679b9ced6c25c: Status 404 returned error can't find the container with id 7bb7ac40385d916826dd093df6f16155191667976900d80110f679b9ced6c25c Nov 03 02:03:03.137310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:03.137289 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc"] Nov 03 02:03:03.353881 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:03.353714 5463 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 03 02:03:03.353881 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:03.353773 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" podUID="88462923-311f-4302-9503-b1dda3e244e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 03 02:03:03.526371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:03.526285 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc" event={"ID":"fcc529d1-9757-467a-bde9-c0afa4822017","Type":"ContainerStarted","Data":"0fefd49e5caa378958ce5bef3a9dc5bdafe9dd5ba2e5dbbc83daa124e0e69fcd"} Nov 03 02:03:03.526371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:03.526353 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc" Nov 03 02:03:03.526598 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:03.526381 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc" event={"ID":"fcc529d1-9757-467a-bde9-c0afa4822017","Type":"ContainerStarted","Data":"7bb7ac40385d916826dd093df6f16155191667976900d80110f679b9ced6c25c"} Nov 03 02:03:03.528065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:03.528040 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst" event={"ID":"e01f46d8-b6be-4f51-9291-a67a09771b4e","Type":"ContainerStarted","Data":"4f6d5c06c5363b6c00afe8458f67a9b558fe721d19f64abd9563c44bea0c6e5f"} Nov 03 02:03:03.528161 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:03.528083 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst" event={"ID":"e01f46d8-b6be-4f51-9291-a67a09771b4e","Type":"ContainerStarted","Data":"8b80bdc8d6b88376889b4b8c82b0e3099588ead75dc18ff750003bdbb9a6126b"} Nov 03 02:03:03.528334 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:03.528318 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst" Nov 03 02:03:03.643664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:03.643616 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc" podStartSLOduration=1.643601809 podStartE2EDuration="1.643601809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:03.641687276 +0000 UTC m=+665.325796431" watchObservedRunningTime="2026-11-03 02:03:03.643601809 +0000 UTC m=+665.327710964" Nov 03 02:03:04.357217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:04.351568 5463 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 03 02:03:04.357217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:04.351627 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" podUID="88462923-311f-4302-9503-b1dda3e244e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 03 02:03:04.433874 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:04.433847 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7" Nov 03 02:03:04.549786 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:04.549732 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst" podStartSLOduration=2.5497136129999998 podStartE2EDuration="2.549713613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:03.775508772 +0000 UTC m=+665.459617931" watchObservedRunningTime="2026-11-03 02:03:04.549713613 +0000 UTC m=+666.233822769" Nov 03 02:03:05.129438 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:05.129409 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:03:05.563503 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:05.563470 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:03:05.564730 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:05.564705 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 02:03:06.033091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:06.033053 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:07.070475 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:07.070436 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:07.220573 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:07.220533 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:07.786553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:07.786523 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:07.948286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:07.948205 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:08.102937 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:08.102852 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:08.431253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:08.431219 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:08.561988 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:08.561953 5463 generic.go:356] "Generic (PLEG): container finished" podID="831449be-effc-4efe-8005-f53a501168a1" containerID="4533b5687e519f60e97c2829a3c3aa50ae280db490914a70217ea31ef604248b" exitCode=0 Nov 03 02:03:08.561988 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:08.561986 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" event={"ID":"831449be-effc-4efe-8005-f53a501168a1","Type":"ContainerDied","Data":"4533b5687e519f60e97c2829a3c3aa50ae280db490914a70217ea31ef604248b"} Nov 03 02:03:08.562385 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:08.562368 5463 scope.go:122] "RemoveContainer" containerID="4533b5687e519f60e97c2829a3c3aa50ae280db490914a70217ea31ef604248b" Nov 03 02:03:08.563754 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:08.563726 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:03:08.563816 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:08.563797 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:03:08.563891 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:08.563831 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:03:08.598669 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:08.598619 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:09.578096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:09.578063 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" event={"ID":"831449be-effc-4efe-8005-f53a501168a1","Type":"ContainerStarted","Data":"e10202ed14343c408c1075548344dae7c28c27aaafcc7273f275135109c4cbc2"} Nov 03 02:03:09.644907 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:09.644877 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:03:09.645116 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:09.645093 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 02:03:09.645235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:09.645147 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" podUID="831449be-effc-4efe-8005-f53a501168a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 02:03:10.645438 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:10.645403 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 02:03:10.645829 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:10.645456 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" podUID="831449be-effc-4efe-8005-f53a501168a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 02:03:11.210498 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:11.210459 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:11.372864 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:11.372828 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:11.523494 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:11.523414 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:11.645513 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:11.645487 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 02:03:11.645947 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:11.645922 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" podUID="831449be-effc-4efe-8005-f53a501168a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 02:03:12.141739 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:12.141704 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:12.645406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:12.645372 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 02:03:12.645591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:12.645423 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" podUID="831449be-effc-4efe-8005-f53a501168a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 02:03:13.008063 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:03:13.007980 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:03:13.644945 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:13.644908 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 02:03:13.645123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:13.644962 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" podUID="831449be-effc-4efe-8005-f53a501168a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 02:03:14.537155 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:14.537126 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst" Nov 03 02:03:14.537582 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:14.537286 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc" Nov 03 02:03:14.663537 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:14.663504 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:03:14.664812 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:14.664701 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:03:14.666451 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:14.666283 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:03:16.123440 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:16.123399 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:20.841484 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:20.841436 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:24.084017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:24.083966 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:24.701678 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:24.701620 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:24.855421 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:24.855388 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:25.019543 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:25.019508 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:25.825921 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:25.825886 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:25.988302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:25.988258 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:26.137361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:26.137271 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:28.715718 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:28.715690 5463 generic.go:356] "Generic (PLEG): container finished" podID="2746f922-b469-4383-90bb-3be08f9256d7" containerID="4665c8b21421aec75f704c6ad2b2457e3cf9c6bfd07b05cf5a00877084b195e2" exitCode=0 Nov 03 02:03:28.716147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:28.715762 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" event={"ID":"2746f922-b469-4383-90bb-3be08f9256d7","Type":"ContainerDied","Data":"4665c8b21421aec75f704c6ad2b2457e3cf9c6bfd07b05cf5a00877084b195e2"} Nov 03 02:03:28.716273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:28.716154 5463 scope.go:122] "RemoveContainer" containerID="4665c8b21421aec75f704c6ad2b2457e3cf9c6bfd07b05cf5a00877084b195e2" Nov 03 02:03:28.717409 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:28.717379 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:03:28.717521 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:28.717459 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:03:28.717521 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:28.717505 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:03:29.723280 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:29.723249 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" event={"ID":"2746f922-b469-4383-90bb-3be08f9256d7","Type":"ContainerStarted","Data":"cb15eb4b66f3990c6f574a0ab10364eee39b9a466f6aeaa245dbdd7e1dc8942c"} Nov 03 02:03:30.597966 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:30.597931 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:03:30.598948 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:30.598924 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 02:03:30.599033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:30.598975 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" podUID="2746f922-b469-4383-90bb-3be08f9256d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 02:03:31.222895 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:31.222863 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:31.365271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:31.358298 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:03:31.597276 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:31.597242 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 02:03:31.597461 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:31.597297 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" podUID="2746f922-b469-4383-90bb-3be08f9256d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 02:03:32.597263 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:32.597226 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 02:03:32.597673 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:32.597293 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" podUID="2746f922-b469-4383-90bb-3be08f9256d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 02:03:33.597661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:33.597626 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 02:03:33.598125 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:33.597681 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" podUID="2746f922-b469-4383-90bb-3be08f9256d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 02:03:34.117495 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:34.117464 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:34.597908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:34.597878 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 02:03:34.598323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:34.597927 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" podUID="2746f922-b469-4383-90bb-3be08f9256d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 02:03:35.601134 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:35.601103 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:03:35.602028 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:35.601998 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:03:36.017524 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:36.017489 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:38.621509 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:38.621476 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:40.831838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:40.831802 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:43.011274 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:03:43.011231 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:03:43.493359 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:43.493322 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:03:43.493556 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:43.493391 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" podUID="e094c575-bfcd-41a1-902a-a348adde1c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:03:45.534266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:45.534121 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:45.832180 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:45.832097 5463 generic.go:356] "Generic (PLEG): container finished" podID="e094c575-bfcd-41a1-902a-a348adde1c14" containerID="628f71f8b1f9ef99eb0408f733671b4fad509b7d267d2d5d1672b3ae54f2c8c8" exitCode=0 Nov 03 02:03:45.832180 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:45.832163 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" event={"ID":"e094c575-bfcd-41a1-902a-a348adde1c14","Type":"ContainerDied","Data":"628f71f8b1f9ef99eb0408f733671b4fad509b7d267d2d5d1672b3ae54f2c8c8"} Nov 03 02:03:45.832583 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:45.832569 5463 scope.go:122] "RemoveContainer" containerID="628f71f8b1f9ef99eb0408f733671b4fad509b7d267d2d5d1672b3ae54f2c8c8" Nov 03 02:03:45.833911 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:45.833885 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:03:45.833966 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:45.833953 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:03:45.834011 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:45.833987 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:03:46.841652 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:46.841613 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" event={"ID":"e094c575-bfcd-41a1-902a-a348adde1c14","Type":"ContainerStarted","Data":"8a937aee1f9c978e6a005b16707d455d10c831ecd5a4820b0b8dcfbd8f41337e"} Nov 03 02:03:47.501459 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:47.501420 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:03:47.501693 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:47.501671 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 02:03:47.501774 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:47.501725 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" podUID="e094c575-bfcd-41a1-902a-a348adde1c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 02:03:48.280391 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:48.280358 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:48.501736 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:48.501701 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 02:03:48.501909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:48.501749 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" podUID="e094c575-bfcd-41a1-902a-a348adde1c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 02:03:48.972251 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:48.972178 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:49.133078 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:49.133046 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:49.502018 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:49.501979 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 02:03:49.502472 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:49.502038 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" podUID="e094c575-bfcd-41a1-902a-a348adde1c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 02:03:50.501593 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:50.501555 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 02:03:50.501771 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:50.501618 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" podUID="e094c575-bfcd-41a1-902a-a348adde1c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 02:03:50.630566 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:50.630534 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:51.501504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:51.501468 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 02:03:51.501700 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:51.501523 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" podUID="e094c575-bfcd-41a1-902a-a348adde1c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 02:03:52.504928 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:52.504896 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:03:52.505424 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:52.505306 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:03:52.506051 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:52.506032 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:03:55.321344 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:03:55.321309 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:00.029512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:00.029480 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:03.968251 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:03.968209 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:04.571200 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:04.571158 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:05.045000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:05.044972 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:05.202864 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:05.202798 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:05.362828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:05.362732 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:07.375535 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:07.375464 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:09.101823 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.101735 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2"] Nov 03 02:04:09.105076 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.105058 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:09.158730 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.158697 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2"] Nov 03 02:04:09.171230 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.171175 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-model-cache\") pod \"simulator-w4-c7d758-kserve-85f59d8fc-8jll2\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:09.171399 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.171245 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e2af92a-6565-45d7-ae87-ef5809aa8517-tls-certs\") pod \"simulator-w4-c7d758-kserve-85f59d8fc-8jll2\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:09.171399 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.171332 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-dshm\") pod \"simulator-w4-c7d758-kserve-85f59d8fc-8jll2\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:09.171399 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.171391 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-tmp-dir\") pod \"simulator-w4-c7d758-kserve-85f59d8fc-8jll2\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:09.171575 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.171422 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcc6q\" (UniqueName: \"kubernetes.io/projected/3e2af92a-6565-45d7-ae87-ef5809aa8517-kube-api-access-gcc6q\") pod \"simulator-w4-c7d758-kserve-85f59d8fc-8jll2\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:09.171575 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.171522 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-home\") pod \"simulator-w4-c7d758-kserve-85f59d8fc-8jll2\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:09.206111 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.206063 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c7d758\"/\"openshift-service-ca.crt\"" Nov 03 02:04:09.206294 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.206216 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c7d758\"/\"simulator-w4-c7d758-kserve-self-signed-certs\"" Nov 03 02:04:09.206294 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.206234 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c7d758\"/\"kube-root-ca.crt\"" Nov 03 02:04:09.207906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.207876 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c7d758\"/\"default-dockercfg-7k4bc\"" Nov 03 02:04:09.214625 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.214600 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:09.272531 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.272499 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-model-cache\") pod \"simulator-w4-c7d758-kserve-85f59d8fc-8jll2\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:09.272720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.272537 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e2af92a-6565-45d7-ae87-ef5809aa8517-tls-certs\") pod \"simulator-w4-c7d758-kserve-85f59d8fc-8jll2\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:09.272720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.272580 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-dshm\") pod \"simulator-w4-c7d758-kserve-85f59d8fc-8jll2\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:09.272720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.272616 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-tmp-dir\") pod \"simulator-w4-c7d758-kserve-85f59d8fc-8jll2\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:09.272720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.272649 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcc6q\" (UniqueName: \"kubernetes.io/projected/3e2af92a-6565-45d7-ae87-ef5809aa8517-kube-api-access-gcc6q\") pod \"simulator-w4-c7d758-kserve-85f59d8fc-8jll2\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:09.272720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.272698 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-home\") pod \"simulator-w4-c7d758-kserve-85f59d8fc-8jll2\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:09.273073 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.273000 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-model-cache\") pod \"simulator-w4-c7d758-kserve-85f59d8fc-8jll2\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:09.273120 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.273077 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-home\") pod \"simulator-w4-c7d758-kserve-85f59d8fc-8jll2\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:09.273311 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.273286 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-tmp-dir\") pod \"simulator-w4-c7d758-kserve-85f59d8fc-8jll2\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:09.275181 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.275157 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-dshm\") pod \"simulator-w4-c7d758-kserve-85f59d8fc-8jll2\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:09.275322 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.275302 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e2af92a-6565-45d7-ae87-ef5809aa8517-tls-certs\") pod \"simulator-w4-c7d758-kserve-85f59d8fc-8jll2\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:09.331161 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.331129 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcc6q\" (UniqueName: \"kubernetes.io/projected/3e2af92a-6565-45d7-ae87-ef5809aa8517-kube-api-access-gcc6q\") pod \"simulator-w4-c7d758-kserve-85f59d8fc-8jll2\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:09.419017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.418977 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:09.796346 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:04:09.796313 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e2af92a_6565_45d7_ae87_ef5809aa8517.slice/crio-957f1ccc33afcb51a8754961cacac1fbfdba59cae0c3d5ade6d2d137846c73e1 WatchSource:0}: Error finding container 957f1ccc33afcb51a8754961cacac1fbfdba59cae0c3d5ade6d2d137846c73e1: Status 404 returned error can't find the container with id 957f1ccc33afcb51a8754961cacac1fbfdba59cae0c3d5ade6d2d137846c73e1 Nov 03 02:04:09.798014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.797967 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2"] Nov 03 02:04:09.858972 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:09.858947 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:10.009548 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:10.009516 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" event={"ID":"3e2af92a-6565-45d7-ae87-ef5809aa8517","Type":"ContainerStarted","Data":"cabe768d8d0d50ac38c42d8e4a8c830de5671354ea82a7ca547f11f1601e6c4e"} Nov 03 02:04:10.009712 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:10.009558 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" event={"ID":"3e2af92a-6565-45d7-ae87-ef5809aa8517","Type":"ContainerStarted","Data":"957f1ccc33afcb51a8754961cacac1fbfdba59cae0c3d5ade6d2d137846c73e1"} Nov 03 02:04:10.124725 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:10.124633 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podStartSLOduration=1.12461861 podStartE2EDuration="1.12461861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:10.122960363 +0000 UTC m=+731.807069518" watchObservedRunningTime="2026-11-03 02:04:10.12461861 +0000 UTC m=+731.808727793" Nov 03 02:04:10.419895 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:10.419866 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:11.482947 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:11.482914 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:13.007017 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:04:13.006980 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:04:14.523335 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:14.523305 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:19.267991 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:19.267954 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:19.419252 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:19.419221 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:19.432157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:19.432129 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:20.086252 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:20.086225 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:04:23.936634 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:23.936598 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:28.623325 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:28.623292 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:30.161997 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:30.161947 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:33.419655 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:33.419618 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:37.510288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.510254 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr"] Nov 03 02:04:37.514449 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.514426 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:37.566855 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.566821 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr"] Nov 03 02:04:37.589657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.589631 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-dshm\") pod \"premium-w1-1bccbd-kserve-699db8f55b-mkkkr\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:37.589657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.589663 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/544033d3-9c0c-4e56-864c-54fcf9d69d7e-tls-certs\") pod \"premium-w1-1bccbd-kserve-699db8f55b-mkkkr\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:37.589881 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.589696 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-home\") pod \"premium-w1-1bccbd-kserve-699db8f55b-mkkkr\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:37.589881 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.589856 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qw4q\" (UniqueName: \"kubernetes.io/projected/544033d3-9c0c-4e56-864c-54fcf9d69d7e-kube-api-access-8qw4q\") pod \"premium-w1-1bccbd-kserve-699db8f55b-mkkkr\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:37.589997 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.589902 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-tmp-dir\") pod \"premium-w1-1bccbd-kserve-699db8f55b-mkkkr\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:37.589997 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.589987 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-model-cache\") pod \"premium-w1-1bccbd-kserve-699db8f55b-mkkkr\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:37.614250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.614224 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1bccbd\"/\"openshift-service-ca.crt\"" Nov 03 02:04:37.614419 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.614308 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1bccbd\"/\"kube-root-ca.crt\"" Nov 03 02:04:37.614419 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.614328 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1bccbd\"/\"default-dockercfg-zdpps\"" Nov 03 02:04:37.617281 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.617260 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:37.617612 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.617594 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1bccbd\"/\"premium-w1-1bccbd-kserve-self-signed-certs\"" Nov 03 02:04:37.690664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.690629 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-dshm\") pod \"premium-w1-1bccbd-kserve-699db8f55b-mkkkr\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:37.690664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.690663 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/544033d3-9c0c-4e56-864c-54fcf9d69d7e-tls-certs\") pod \"premium-w1-1bccbd-kserve-699db8f55b-mkkkr\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:37.690922 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.690704 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-home\") pod \"premium-w1-1bccbd-kserve-699db8f55b-mkkkr\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:37.690922 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.690787 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qw4q\" (UniqueName: \"kubernetes.io/projected/544033d3-9c0c-4e56-864c-54fcf9d69d7e-kube-api-access-8qw4q\") pod \"premium-w1-1bccbd-kserve-699db8f55b-mkkkr\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:37.690922 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.690832 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-tmp-dir\") pod \"premium-w1-1bccbd-kserve-699db8f55b-mkkkr\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:37.690922 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.690902 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-model-cache\") pod \"premium-w1-1bccbd-kserve-699db8f55b-mkkkr\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:37.691248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.691222 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-home\") pod \"premium-w1-1bccbd-kserve-699db8f55b-mkkkr\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:37.691333 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.691284 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-model-cache\") pod \"premium-w1-1bccbd-kserve-699db8f55b-mkkkr\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:37.691434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.691414 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-tmp-dir\") pod \"premium-w1-1bccbd-kserve-699db8f55b-mkkkr\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:37.693057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.693033 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-dshm\") pod \"premium-w1-1bccbd-kserve-699db8f55b-mkkkr\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:37.693267 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.693245 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/544033d3-9c0c-4e56-864c-54fcf9d69d7e-tls-certs\") pod \"premium-w1-1bccbd-kserve-699db8f55b-mkkkr\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:37.747404 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.747377 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qw4q\" (UniqueName: \"kubernetes.io/projected/544033d3-9c0c-4e56-864c-54fcf9d69d7e-kube-api-access-8qw4q\") pod \"premium-w1-1bccbd-kserve-699db8f55b-mkkkr\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:37.827732 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:37.827666 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:38.222526 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:04:38.222489 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod544033d3_9c0c_4e56_864c_54fcf9d69d7e.slice/crio-6410b07bb06ed2a3593a520419eafd2c7b407c42872a0b693da9db4d4d850881 WatchSource:0}: Error finding container 6410b07bb06ed2a3593a520419eafd2c7b407c42872a0b693da9db4d4d850881: Status 404 returned error can't find the container with id 6410b07bb06ed2a3593a520419eafd2c7b407c42872a0b693da9db4d4d850881 Nov 03 02:04:38.225494 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:38.225475 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr"] Nov 03 02:04:38.229045 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:38.229024 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:39.225599 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:39.225564 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" event={"ID":"544033d3-9c0c-4e56-864c-54fcf9d69d7e","Type":"ContainerStarted","Data":"2ffdf6d388c8da0fdeffe1f2ad07410354e18a383ecd90b009bc71408b72d047"} Nov 03 02:04:39.226065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:39.225608 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" event={"ID":"544033d3-9c0c-4e56-864c-54fcf9d69d7e","Type":"ContainerStarted","Data":"6410b07bb06ed2a3593a520419eafd2c7b407c42872a0b693da9db4d4d850881"} Nov 03 02:04:39.341430 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:39.341363 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podStartSLOduration=2.341342819 podStartE2EDuration="2.341342819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:39.339746136 +0000 UTC m=+761.023855291" watchObservedRunningTime="2026-11-03 02:04:39.341342819 +0000 UTC m=+761.025451976" Nov 03 02:04:39.828396 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:39.828356 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:41.537547 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:41.537519 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:43.006254 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:04:43.006217 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:04:43.032522 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:43.032490 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:47.821004 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:47.820968 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:47.827933 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:47.827906 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:47.840284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:47.840265 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:48.295500 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:48.295466 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:04:52.535311 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:52.535279 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:57.205920 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:04:57.205878 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:01.923054 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:01.923015 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:06.724748 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:06.724692 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:07.696165 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:07.696129 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:11.432228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.432178 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:11.700443 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.700361 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q"] Nov 03 02:05:11.705363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.705348 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:11.767265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.767220 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-model-cache\") pod \"premium-w4-c7d758-kserve-748696cd88-d442q\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:11.767452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.767331 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-home\") pod \"premium-w4-c7d758-kserve-748696cd88-d442q\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:11.767452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.767387 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-dshm\") pod \"premium-w4-c7d758-kserve-748696cd88-d442q\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:11.767452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.767412 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2f08835-5113-4ca7-9cfa-e107685e3348-tls-certs\") pod \"premium-w4-c7d758-kserve-748696cd88-d442q\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:11.767639 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.767501 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98szb\" (UniqueName: \"kubernetes.io/projected/e2f08835-5113-4ca7-9cfa-e107685e3348-kube-api-access-98szb\") pod \"premium-w4-c7d758-kserve-748696cd88-d442q\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:11.767639 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.767540 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-tmp-dir\") pod \"premium-w4-c7d758-kserve-748696cd88-d442q\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:11.808545 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.808516 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c7d758\"/\"premium-w4-c7d758-kserve-self-signed-certs\"" Nov 03 02:05:11.812024 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.811992 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q"] Nov 03 02:05:11.823090 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.823058 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:11.869046 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.869005 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-dshm\") pod \"premium-w4-c7d758-kserve-748696cd88-d442q\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:11.869259 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.869063 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2f08835-5113-4ca7-9cfa-e107685e3348-tls-certs\") pod \"premium-w4-c7d758-kserve-748696cd88-d442q\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:11.869259 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.869131 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98szb\" (UniqueName: \"kubernetes.io/projected/e2f08835-5113-4ca7-9cfa-e107685e3348-kube-api-access-98szb\") pod \"premium-w4-c7d758-kserve-748696cd88-d442q\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:11.869259 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.869206 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-tmp-dir\") pod \"premium-w4-c7d758-kserve-748696cd88-d442q\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:11.869435 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.869328 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-model-cache\") pod \"premium-w4-c7d758-kserve-748696cd88-d442q\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:11.869629 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.869602 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-tmp-dir\") pod \"premium-w4-c7d758-kserve-748696cd88-d442q\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:11.869686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.869657 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-model-cache\") pod \"premium-w4-c7d758-kserve-748696cd88-d442q\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:11.869811 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.869781 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-home\") pod \"premium-w4-c7d758-kserve-748696cd88-d442q\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:11.871223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.870147 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-home\") pod \"premium-w4-c7d758-kserve-748696cd88-d442q\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:11.873708 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.873686 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-dshm\") pod \"premium-w4-c7d758-kserve-748696cd88-d442q\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:11.875038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.875015 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2f08835-5113-4ca7-9cfa-e107685e3348-tls-certs\") pod \"premium-w4-c7d758-kserve-748696cd88-d442q\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:11.936588 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:11.936521 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98szb\" (UniqueName: \"kubernetes.io/projected/e2f08835-5113-4ca7-9cfa-e107685e3348-kube-api-access-98szb\") pod \"premium-w4-c7d758-kserve-748696cd88-d442q\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:12.019463 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:12.019380 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:12.596024 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:05:12.595966 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2f08835_5113_4ca7_9cfa_e107685e3348.slice/crio-4fd161cb13017f7ddc41eb65b542d0b4ba5df370cff9051405d6939db971fe9e WatchSource:0}: Error finding container 4fd161cb13017f7ddc41eb65b542d0b4ba5df370cff9051405d6939db971fe9e: Status 404 returned error can't find the container with id 4fd161cb13017f7ddc41eb65b542d0b4ba5df370cff9051405d6939db971fe9e Nov 03 02:05:12.599342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:12.599302 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q"] Nov 03 02:05:13.007268 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:05:13.007233 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:05:13.453071 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:13.453038 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" event={"ID":"e2f08835-5113-4ca7-9cfa-e107685e3348","Type":"ContainerStarted","Data":"d48027a489669f17801f1913a3155d0898ef687f1cccfa94f109b7e6177de1f0"} Nov 03 02:05:13.453254 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:13.453080 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" event={"ID":"e2f08835-5113-4ca7-9cfa-e107685e3348","Type":"ContainerStarted","Data":"4fd161cb13017f7ddc41eb65b542d0b4ba5df370cff9051405d6939db971fe9e"} Nov 03 02:05:13.568774 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:13.568717 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podStartSLOduration=2.5686993129999998 podStartE2EDuration="2.568699313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:05:13.566723989 +0000 UTC m=+795.250833145" watchObservedRunningTime="2026-11-03 02:05:13.568699313 +0000 UTC m=+795.252808469" Nov 03 02:05:14.020264 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:14.020220 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:16.093709 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:16.093680 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:18.422011 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:18.421969 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568"] Nov 03 02:05:18.426117 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:18.426095 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568" Nov 03 02:05:18.479432 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:18.479400 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568"] Nov 03 02:05:18.526083 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:18.526050 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:05:18.528954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:18.528930 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:05:18.528954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:18.528944 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:05:18.529081 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:18.529005 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:05:18.549267 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:18.549234 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvfjq\" (UniqueName: \"kubernetes.io/projected/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-kube-api-access-qvfjq\") pod \"maas-api-revoke-sub-f3783ebd-9w568\" (UID: \"1e3bab4f-46b5-4a0f-b16a-adc88af640c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568" Nov 03 02:05:18.549433 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:18.549341 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3783ebd-9w568\" (UID: \"1e3bab4f-46b5-4a0f-b16a-adc88af640c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568" Nov 03 02:05:18.549498 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:18.549459 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3783ebd-9w568\" (UID: \"1e3bab4f-46b5-4a0f-b16a-adc88af640c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568" Nov 03 02:05:18.650434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:18.650406 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvfjq\" (UniqueName: \"kubernetes.io/projected/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-kube-api-access-qvfjq\") pod \"maas-api-revoke-sub-f3783ebd-9w568\" (UID: \"1e3bab4f-46b5-4a0f-b16a-adc88af640c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568" Nov 03 02:05:18.650619 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:18.650485 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3783ebd-9w568\" (UID: \"1e3bab4f-46b5-4a0f-b16a-adc88af640c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568" Nov 03 02:05:18.650619 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:18.650539 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3783ebd-9w568\" (UID: \"1e3bab4f-46b5-4a0f-b16a-adc88af640c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568" Nov 03 02:05:18.651149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:18.651117 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3783ebd-9w568\" (UID: \"1e3bab4f-46b5-4a0f-b16a-adc88af640c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568" Nov 03 02:05:18.652840 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:18.652816 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3783ebd-9w568\" (UID: \"1e3bab4f-46b5-4a0f-b16a-adc88af640c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568" Nov 03 02:05:18.708443 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:18.708373 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvfjq\" (UniqueName: \"kubernetes.io/projected/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-kube-api-access-qvfjq\") pod \"maas-api-revoke-sub-f3783ebd-9w568\" (UID: \"1e3bab4f-46b5-4a0f-b16a-adc88af640c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568" Nov 03 02:05:18.740816 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:18.740789 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568" Nov 03 02:05:19.127117 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:05:19.127064 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3bab4f_46b5_4a0f_b16a_adc88af640c7.slice/crio-7f19764a3d0788ca8902a67f98561e45facebecbad568f9df2758a92486abfc0 WatchSource:0}: Error finding container 7f19764a3d0788ca8902a67f98561e45facebecbad568f9df2758a92486abfc0: Status 404 returned error can't find the container with id 7f19764a3d0788ca8902a67f98561e45facebecbad568f9df2758a92486abfc0 Nov 03 02:05:19.130442 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:19.130410 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568"] Nov 03 02:05:19.515292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:19.515258 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568" event={"ID":"1e3bab4f-46b5-4a0f-b16a-adc88af640c7","Type":"ContainerStarted","Data":"95aec74faf3f39e2bebf5261b76aabf6c934a2d99fc2749bda1c02e3bc133379"} Nov 03 02:05:19.515635 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:19.515318 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568" event={"ID":"1e3bab4f-46b5-4a0f-b16a-adc88af640c7","Type":"ContainerStarted","Data":"7f19764a3d0788ca8902a67f98561e45facebecbad568f9df2758a92486abfc0"} Nov 03 02:05:19.629387 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:19.629340 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568" podStartSLOduration=1.629327441 podStartE2EDuration="1.629327441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:05:19.628487994 +0000 UTC m=+801.312597150" watchObservedRunningTime="2026-11-03 02:05:19.629327441 +0000 UTC m=+801.313436637" Nov 03 02:05:20.523900 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:20.523866 5463 generic.go:356] "Generic (PLEG): container finished" podID="1e3bab4f-46b5-4a0f-b16a-adc88af640c7" containerID="95aec74faf3f39e2bebf5261b76aabf6c934a2d99fc2749bda1c02e3bc133379" exitCode=0 Nov 03 02:05:20.524332 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:20.523954 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568" event={"ID":"1e3bab4f-46b5-4a0f-b16a-adc88af640c7","Type":"ContainerDied","Data":"95aec74faf3f39e2bebf5261b76aabf6c934a2d99fc2749bda1c02e3bc133379"} Nov 03 02:05:20.822810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:20.822720 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:21.684097 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:21.684074 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568" Nov 03 02:05:21.804338 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:21.804266 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-maas-api-xks-ca\") pod \"1e3bab4f-46b5-4a0f-b16a-adc88af640c7\" (UID: \"1e3bab4f-46b5-4a0f-b16a-adc88af640c7\") " Nov 03 02:05:21.804338 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:21.804309 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-kube-api-access-qvfjq\" (UniqueName: \"kubernetes.io/projected/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-kube-api-access-qvfjq\") pod \"1e3bab4f-46b5-4a0f-b16a-adc88af640c7\" (UID: \"1e3bab4f-46b5-4a0f-b16a-adc88af640c7\") " Nov 03 02:05:21.804539 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:21.804481 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-maas-api-openshift-ca\") pod \"1e3bab4f-46b5-4a0f-b16a-adc88af640c7\" (UID: \"1e3bab4f-46b5-4a0f-b16a-adc88af640c7\") " Nov 03 02:05:21.804974 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:21.804786 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-maas-api-openshift-ca" pod "1e3bab4f-46b5-4a0f-b16a-adc88af640c7" (UID: "1e3bab4f-46b5-4a0f-b16a-adc88af640c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:05:21.804974 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:21.804949 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:21.806518 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:21.806481 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-kube-api-access-qvfjq" pod "1e3bab4f-46b5-4a0f-b16a-adc88af640c7" (UID: "1e3bab4f-46b5-4a0f-b16a-adc88af640c7"). InnerVolumeSpecName "kube-api-access-qvfjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:05:21.806979 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:21.806958 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-maas-api-xks-ca" pod "1e3bab4f-46b5-4a0f-b16a-adc88af640c7" (UID: "1e3bab4f-46b5-4a0f-b16a-adc88af640c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:05:21.905897 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:21.905873 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:21.905897 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:21.905894 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvfjq\" (UniqueName: \"kubernetes.io/projected/1e3bab4f-46b5-4a0f-b16a-adc88af640c7-kube-api-access-qvfjq\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:22.020711 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:22.020685 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:22.035211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:22.034840 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:22.536744 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:22.536715 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568" Nov 03 02:05:22.536887 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:22.536747 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568" event={"ID":"1e3bab4f-46b5-4a0f-b16a-adc88af640c7","Type":"ContainerDied","Data":"7f19764a3d0788ca8902a67f98561e45facebecbad568f9df2758a92486abfc0"} Nov 03 02:05:22.536887 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:22.536796 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f19764a3d0788ca8902a67f98561e45facebecbad568f9df2758a92486abfc0" Nov 03 02:05:22.548527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:22.548503 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:05:25.531571 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:25.531531 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:30.225793 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:30.225754 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:34.904626 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:34.904592 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq"] Nov 03 02:05:34.905103 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:34.904800 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" podUID="dab070df-285a-4b9f-8d66-b0b6f976cecb" containerName="istio-proxy" containerID="cri-o://ac3e6c0512defa4a78c01cc97b784c8c39a6ebf0d5c85dbf225070f291e07e06" gracePeriod=30 Nov 03 02:05:34.935864 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:34.935831 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:35.015063 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.015031 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8"] Nov 03 02:05:35.016377 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.016347 5463 cpu_manager.go:414] "containerMap: removing container" podUID="1e3bab4f-46b5-4a0f-b16a-adc88af640c7" containerName="revoke-keys" Nov 03 02:05:35.016864 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.016815 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e3bab4f-46b5-4a0f-b16a-adc88af640c7" containerName="revoke-keys" Nov 03 02:05:35.017057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.017036 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e3bab4f-46b5-4a0f-b16a-adc88af640c7" containerName="revoke-keys" Nov 03 02:05:35.023313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.023289 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8" Nov 03 02:05:35.069919 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.069888 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8"] Nov 03 02:05:35.076521 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.076496 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:05:35.076687 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.076665 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:05:35.076750 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.076626 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:05:35.076750 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.076504 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:05:35.149563 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.149539 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb5a7e61-vvkd8\" (UID: \"31abc5f5-336f-4f7c-ba26-380a23e8ab9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8" Nov 03 02:05:35.149743 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.149722 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zmv9\" (UniqueName: \"kubernetes.io/projected/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-kube-api-access-6zmv9\") pod \"maas-api-revoke-sub-eb5a7e61-vvkd8\" (UID: \"31abc5f5-336f-4f7c-ba26-380a23e8ab9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8" Nov 03 02:05:35.149788 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.149763 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb5a7e61-vvkd8\" (UID: \"31abc5f5-336f-4f7c-ba26-380a23e8ab9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8" Nov 03 02:05:35.251403 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.251322 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zmv9\" (UniqueName: \"kubernetes.io/projected/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-kube-api-access-6zmv9\") pod \"maas-api-revoke-sub-eb5a7e61-vvkd8\" (UID: \"31abc5f5-336f-4f7c-ba26-380a23e8ab9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8" Nov 03 02:05:35.251403 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.251365 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb5a7e61-vvkd8\" (UID: \"31abc5f5-336f-4f7c-ba26-380a23e8ab9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8" Nov 03 02:05:35.251612 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.251497 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb5a7e61-vvkd8\" (UID: \"31abc5f5-336f-4f7c-ba26-380a23e8ab9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8" Nov 03 02:05:35.251957 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.251936 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb5a7e61-vvkd8\" (UID: \"31abc5f5-336f-4f7c-ba26-380a23e8ab9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8" Nov 03 02:05:35.253867 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.253846 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb5a7e61-vvkd8\" (UID: \"31abc5f5-336f-4f7c-ba26-380a23e8ab9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8" Nov 03 02:05:35.308421 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.308395 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zmv9\" (UniqueName: \"kubernetes.io/projected/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-kube-api-access-6zmv9\") pod \"maas-api-revoke-sub-eb5a7e61-vvkd8\" (UID: \"31abc5f5-336f-4f7c-ba26-380a23e8ab9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8" Nov 03 02:05:35.367492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.367455 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8" Nov 03 02:05:35.733941 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:05:35.733910 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31abc5f5_336f_4f7c_ba26_380a23e8ab9e.slice/crio-fd666c1072be0bd5ae2e6760070b56c76c0d0e0ec11ecbd771d08659fdc8e643 WatchSource:0}: Error finding container fd666c1072be0bd5ae2e6760070b56c76c0d0e0ec11ecbd771d08659fdc8e643: Status 404 returned error can't find the container with id fd666c1072be0bd5ae2e6760070b56c76c0d0e0ec11ecbd771d08659fdc8e643 Nov 03 02:05:35.735933 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:35.735908 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8"] Nov 03 02:05:36.645759 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:36.645727 5463 generic.go:356] "Generic (PLEG): container finished" podID="31abc5f5-336f-4f7c-ba26-380a23e8ab9e" containerID="f0a8d53788bf459a5e7830e8adeddc276e02e2ee2a7ed8099fd8313ffe71398a" exitCode=0 Nov 03 02:05:36.646145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:36.645765 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8" event={"ID":"31abc5f5-336f-4f7c-ba26-380a23e8ab9e","Type":"ContainerDied","Data":"f0a8d53788bf459a5e7830e8adeddc276e02e2ee2a7ed8099fd8313ffe71398a"} Nov 03 02:05:36.646145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:36.645818 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8" event={"ID":"31abc5f5-336f-4f7c-ba26-380a23e8ab9e","Type":"ContainerStarted","Data":"fd666c1072be0bd5ae2e6760070b56c76c0d0e0ec11ecbd771d08659fdc8e643"} Nov 03 02:05:37.811575 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:37.811546 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8" Nov 03 02:05:37.881368 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:37.881325 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-kube-api-access-6zmv9\" (UniqueName: \"kubernetes.io/projected/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-kube-api-access-6zmv9\") pod \"31abc5f5-336f-4f7c-ba26-380a23e8ab9e\" (UID: \"31abc5f5-336f-4f7c-ba26-380a23e8ab9e\") " Nov 03 02:05:37.881530 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:37.881394 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-maas-api-xks-ca\") pod \"31abc5f5-336f-4f7c-ba26-380a23e8ab9e\" (UID: \"31abc5f5-336f-4f7c-ba26-380a23e8ab9e\") " Nov 03 02:05:37.881530 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:37.881502 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-maas-api-openshift-ca\") pod \"31abc5f5-336f-4f7c-ba26-380a23e8ab9e\" (UID: \"31abc5f5-336f-4f7c-ba26-380a23e8ab9e\") " Nov 03 02:05:37.881828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:37.881805 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-maas-api-openshift-ca" pod "31abc5f5-336f-4f7c-ba26-380a23e8ab9e" (UID: "31abc5f5-336f-4f7c-ba26-380a23e8ab9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:05:37.881962 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:37.881939 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:37.883570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:37.883550 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-kube-api-access-6zmv9" pod "31abc5f5-336f-4f7c-ba26-380a23e8ab9e" (UID: "31abc5f5-336f-4f7c-ba26-380a23e8ab9e"). InnerVolumeSpecName "kube-api-access-6zmv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:05:37.884015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:37.883992 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-maas-api-xks-ca" pod "31abc5f5-336f-4f7c-ba26-380a23e8ab9e" (UID: "31abc5f5-336f-4f7c-ba26-380a23e8ab9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:05:37.982904 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:37.982813 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zmv9\" (UniqueName: \"kubernetes.io/projected/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-kube-api-access-6zmv9\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:37.982904 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:37.982845 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31abc5f5-336f-4f7c-ba26-380a23e8ab9e-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:38.522697 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.522664 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p"] Nov 03 02:05:38.523262 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.523246 5463 cpu_manager.go:414] "containerMap: removing container" podUID="31abc5f5-336f-4f7c-ba26-380a23e8ab9e" containerName="revoke-keys" Nov 03 02:05:38.523318 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.523264 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31abc5f5-336f-4f7c-ba26-380a23e8ab9e" containerName="revoke-keys" Nov 03 02:05:38.523407 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.523393 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="31abc5f5-336f-4f7c-ba26-380a23e8ab9e" containerName="revoke-keys" Nov 03 02:05:38.526886 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.526867 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:38.588069 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.588034 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-dshm\") pod \"premium-w0-f7d9bb-kserve-7c6c9676db-hws4p\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:38.588287 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.588258 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-tmp-dir\") pod \"premium-w0-f7d9bb-kserve-7c6c9676db-hws4p\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:38.588366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.588318 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-home\") pod \"premium-w0-f7d9bb-kserve-7c6c9676db-hws4p\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:38.588366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.588361 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230d658f-a760-4d30-92f4-b021eda43859-tls-certs\") pod \"premium-w0-f7d9bb-kserve-7c6c9676db-hws4p\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:38.588491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.588395 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm49p\" (UniqueName: \"kubernetes.io/projected/230d658f-a760-4d30-92f4-b021eda43859-kube-api-access-cm49p\") pod \"premium-w0-f7d9bb-kserve-7c6c9676db-hws4p\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:38.588540 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.588489 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-model-cache\") pod \"premium-w0-f7d9bb-kserve-7c6c9676db-hws4p\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:38.597581 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.597551 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f7d9bb\"/\"kube-root-ca.crt\"" Nov 03 02:05:38.597916 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.597867 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f7d9bb\"/\"openshift-service-ca.crt\"" Nov 03 02:05:38.629843 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.629825 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f7d9bb\"/\"premium-w0-f7d9bb-kserve-self-signed-certs\"" Nov 03 02:05:38.630118 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.630099 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f7d9bb\"/\"default-dockercfg-57ngd\"" Nov 03 02:05:38.632713 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.632694 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p"] Nov 03 02:05:38.639353 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.639325 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:38.666481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.666265 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8" Nov 03 02:05:38.666481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.666268 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8" event={"ID":"31abc5f5-336f-4f7c-ba26-380a23e8ab9e","Type":"ContainerDied","Data":"fd666c1072be0bd5ae2e6760070b56c76c0d0e0ec11ecbd771d08659fdc8e643"} Nov 03 02:05:38.666481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.666386 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd666c1072be0bd5ae2e6760070b56c76c0d0e0ec11ecbd771d08659fdc8e643" Nov 03 02:05:38.689161 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.689124 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-model-cache\") pod \"premium-w0-f7d9bb-kserve-7c6c9676db-hws4p\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:38.689395 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.689180 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-dshm\") pod \"premium-w0-f7d9bb-kserve-7c6c9676db-hws4p\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:38.689395 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.689313 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-tmp-dir\") pod \"premium-w0-f7d9bb-kserve-7c6c9676db-hws4p\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:38.689395 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.689355 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-home\") pod \"premium-w0-f7d9bb-kserve-7c6c9676db-hws4p\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:38.689395 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.689393 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230d658f-a760-4d30-92f4-b021eda43859-tls-certs\") pod \"premium-w0-f7d9bb-kserve-7c6c9676db-hws4p\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:38.689611 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.689430 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm49p\" (UniqueName: \"kubernetes.io/projected/230d658f-a760-4d30-92f4-b021eda43859-kube-api-access-cm49p\") pod \"premium-w0-f7d9bb-kserve-7c6c9676db-hws4p\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:38.689892 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.689868 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-model-cache\") pod \"premium-w0-f7d9bb-kserve-7c6c9676db-hws4p\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:38.690391 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.690223 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-tmp-dir\") pod \"premium-w0-f7d9bb-kserve-7c6c9676db-hws4p\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:38.690584 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.690301 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-home\") pod \"premium-w0-f7d9bb-kserve-7c6c9676db-hws4p\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:38.692317 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.692293 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230d658f-a760-4d30-92f4-b021eda43859-tls-certs\") pod \"premium-w0-f7d9bb-kserve-7c6c9676db-hws4p\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:38.692618 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.692598 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-dshm\") pod \"premium-w0-f7d9bb-kserve-7c6c9676db-hws4p\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:38.747000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.746934 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm49p\" (UniqueName: \"kubernetes.io/projected/230d658f-a760-4d30-92f4-b021eda43859-kube-api-access-cm49p\") pod \"premium-w0-f7d9bb-kserve-7c6c9676db-hws4p\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:38.839899 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:38.839869 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:39.242771 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:05:39.242740 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod230d658f_a760_4d30_92f4_b021eda43859.slice/crio-cd93d181cfa62de2babf0515eda3c9a167e628501752dfffdc3b7093bc670abb WatchSource:0}: Error finding container cd93d181cfa62de2babf0515eda3c9a167e628501752dfffdc3b7093bc670abb: Status 404 returned error can't find the container with id cd93d181cfa62de2babf0515eda3c9a167e628501752dfffdc3b7093bc670abb Nov 03 02:05:39.243718 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:39.243695 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p"] Nov 03 02:05:39.629983 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:39.629899 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:39.678936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:39.678895 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" event={"ID":"230d658f-a760-4d30-92f4-b021eda43859","Type":"ContainerStarted","Data":"dc8e9c83ba6d266504f57cf11a83c015bc55e2fa683ad7727a3651368455c96b"} Nov 03 02:05:39.679099 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:39.678947 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" event={"ID":"230d658f-a760-4d30-92f4-b021eda43859","Type":"ContainerStarted","Data":"cd93d181cfa62de2babf0515eda3c9a167e628501752dfffdc3b7093bc670abb"} Nov 03 02:05:39.796406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:39.796347 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podStartSLOduration=1.7963289329999999 podStartE2EDuration="1.796328933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:05:39.792287387 +0000 UTC m=+821.476396543" watchObservedRunningTime="2026-11-03 02:05:39.796328933 +0000 UTC m=+821.480438090" Nov 03 02:05:39.840521 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:39.840489 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:40.203279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.203255 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:05:40.311568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.311528 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dab070df-285a-4b9f-8d66-b0b6f976cecb-kube-api-access-whj66\" (UniqueName: \"kubernetes.io/projected/dab070df-285a-4b9f-8d66-b0b6f976cecb-kube-api-access-whj66\") pod \"dab070df-285a-4b9f-8d66-b0b6f976cecb\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " Nov 03 02:05:40.311762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.311732 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-workload-socket\") pod \"dab070df-285a-4b9f-8d66-b0b6f976cecb\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " Nov 03 02:05:40.311846 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.311767 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-workload-certs\") pod \"dab070df-285a-4b9f-8d66-b0b6f976cecb\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " Nov 03 02:05:40.311846 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.311789 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-credential-socket\") pod \"dab070df-285a-4b9f-8d66-b0b6f976cecb\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " Nov 03 02:05:40.311846 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.311830 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-envoy\") pod \"dab070df-285a-4b9f-8d66-b0b6f976cecb\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " Nov 03 02:05:40.312012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.311872 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-token\" (UniqueName: \"kubernetes.io/projected/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-token\") pod \"dab070df-285a-4b9f-8d66-b0b6f976cecb\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " Nov 03 02:05:40.312012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.311901 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-podinfo\") pod \"dab070df-285a-4b9f-8d66-b0b6f976cecb\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " Nov 03 02:05:40.312012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.311926 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-data\") pod \"dab070df-285a-4b9f-8d66-b0b6f976cecb\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " Nov 03 02:05:40.312012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.311967 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dab070df-285a-4b9f-8d66-b0b6f976cecb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab070df-285a-4b9f-8d66-b0b6f976cecb-istiod-ca-cert\") pod \"dab070df-285a-4b9f-8d66-b0b6f976cecb\" (UID: \"dab070df-285a-4b9f-8d66-b0b6f976cecb\") " Nov 03 02:05:40.312241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.312044 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-workload-socket" pod "dab070df-285a-4b9f-8d66-b0b6f976cecb" (UID: "dab070df-285a-4b9f-8d66-b0b6f976cecb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:05:40.312241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.312100 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-credential-socket" pod "dab070df-285a-4b9f-8d66-b0b6f976cecb" (UID: "dab070df-285a-4b9f-8d66-b0b6f976cecb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:05:40.312461 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.312441 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab070df-285a-4b9f-8d66-b0b6f976cecb-istiod-ca-cert" pod "dab070df-285a-4b9f-8d66-b0b6f976cecb" (UID: "dab070df-285a-4b9f-8d66-b0b6f976cecb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:05:40.312544 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.312464 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:40.312544 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.312482 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:40.312544 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.312488 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-data" pod "dab070df-285a-4b9f-8d66-b0b6f976cecb" (UID: "dab070df-285a-4b9f-8d66-b0b6f976cecb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:05:40.312703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.312561 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-workload-certs" pod "dab070df-285a-4b9f-8d66-b0b6f976cecb" (UID: "dab070df-285a-4b9f-8d66-b0b6f976cecb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:05:40.314303 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.314279 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-token" pod "dab070df-285a-4b9f-8d66-b0b6f976cecb" (UID: "dab070df-285a-4b9f-8d66-b0b6f976cecb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:05:40.314447 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.314383 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab070df-285a-4b9f-8d66-b0b6f976cecb-kube-api-access-whj66" pod "dab070df-285a-4b9f-8d66-b0b6f976cecb" (UID: "dab070df-285a-4b9f-8d66-b0b6f976cecb"). InnerVolumeSpecName "kube-api-access-whj66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:05:40.314723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.314698 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-podinfo" pod "dab070df-285a-4b9f-8d66-b0b6f976cecb" (UID: "dab070df-285a-4b9f-8d66-b0b6f976cecb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:05:40.315092 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.315077 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-envoy" pod "dab070df-285a-4b9f-8d66-b0b6f976cecb" (UID: "dab070df-285a-4b9f-8d66-b0b6f976cecb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:05:40.414012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.413978 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whj66\" (UniqueName: \"kubernetes.io/projected/dab070df-285a-4b9f-8d66-b0b6f976cecb-kube-api-access-whj66\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:40.414012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.414015 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:40.414242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.414027 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:40.414242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.414039 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:40.414242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.414052 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:40.414242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.414064 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab070df-285a-4b9f-8d66-b0b6f976cecb-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:40.414242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.414076 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab070df-285a-4b9f-8d66-b0b6f976cecb-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:40.614778 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.614749 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf"] Nov 03 02:05:40.615336 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.615319 5463 cpu_manager.go:414] "containerMap: removing container" podUID="dab070df-285a-4b9f-8d66-b0b6f976cecb" containerName="istio-proxy" Nov 03 02:05:40.615412 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.615336 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dab070df-285a-4b9f-8d66-b0b6f976cecb" containerName="istio-proxy" Nov 03 02:05:40.615412 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.615409 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="dab070df-285a-4b9f-8d66-b0b6f976cecb" containerName="istio-proxy" Nov 03 02:05:40.618846 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.618828 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf" Nov 03 02:05:40.670399 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.670374 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf"] Nov 03 02:05:40.672451 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.671957 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:05:40.672451 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.672158 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:05:40.672451 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.672419 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:05:40.672451 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.672437 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:05:40.693073 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.693036 5463 generic.go:356] "Generic (PLEG): container finished" podID="dab070df-285a-4b9f-8d66-b0b6f976cecb" containerID="ac3e6c0512defa4a78c01cc97b784c8c39a6ebf0d5c85dbf225070f291e07e06" exitCode=0 Nov 03 02:05:40.693221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.693117 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" event={"ID":"dab070df-285a-4b9f-8d66-b0b6f976cecb","Type":"ContainerDied","Data":"ac3e6c0512defa4a78c01cc97b784c8c39a6ebf0d5c85dbf225070f291e07e06"} Nov 03 02:05:40.693221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.693146 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" Nov 03 02:05:40.693354 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.693271 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq" event={"ID":"dab070df-285a-4b9f-8d66-b0b6f976cecb","Type":"ContainerDied","Data":"a27865e00dd946749c56dab5c3fb151248b8cfe88726f380f721f454b0889719"} Nov 03 02:05:40.693354 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.693309 5463 scope.go:122] "RemoveContainer" containerID="ac3e6c0512defa4a78c01cc97b784c8c39a6ebf0d5c85dbf225070f291e07e06" Nov 03 02:05:40.738215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.732950 5463 scope.go:122] "RemoveContainer" containerID="ac3e6c0512defa4a78c01cc97b784c8c39a6ebf0d5c85dbf225070f291e07e06" Nov 03 02:05:40.738215 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:05:40.737464 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3e6c0512defa4a78c01cc97b784c8c39a6ebf0d5c85dbf225070f291e07e06\": container with ID starting with ac3e6c0512defa4a78c01cc97b784c8c39a6ebf0d5c85dbf225070f291e07e06 not found: ID does not exist" containerID="ac3e6c0512defa4a78c01cc97b784c8c39a6ebf0d5c85dbf225070f291e07e06" Nov 03 02:05:40.738215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.737504 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3e6c0512defa4a78c01cc97b784c8c39a6ebf0d5c85dbf225070f291e07e06"} err="failed to get container status \"ac3e6c0512defa4a78c01cc97b784c8c39a6ebf0d5c85dbf225070f291e07e06\": rpc error: code = NotFound desc = could not find container \"ac3e6c0512defa4a78c01cc97b784c8c39a6ebf0d5c85dbf225070f291e07e06\": container with ID starting with ac3e6c0512defa4a78c01cc97b784c8c39a6ebf0d5c85dbf225070f291e07e06 not found: ID does not exist" Nov 03 02:05:40.840700 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.840660 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6320d42b-af84-46aa-b036-812a3ca9c786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46494c32-j9bxf\" (UID: \"6320d42b-af84-46aa-b036-812a3ca9c786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf" Nov 03 02:05:40.841266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.840885 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hql4x\" (UniqueName: \"kubernetes.io/projected/6320d42b-af84-46aa-b036-812a3ca9c786-kube-api-access-hql4x\") pod \"maas-api-revoke-sub-46494c32-j9bxf\" (UID: \"6320d42b-af84-46aa-b036-812a3ca9c786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf" Nov 03 02:05:40.841266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.840997 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6320d42b-af84-46aa-b036-812a3ca9c786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46494c32-j9bxf\" (UID: \"6320d42b-af84-46aa-b036-812a3ca9c786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf" Nov 03 02:05:40.848700 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.848674 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq"] Nov 03 02:05:40.852722 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.852693 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9f4sq"] Nov 03 02:05:40.942343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.942314 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hql4x\" (UniqueName: \"kubernetes.io/projected/6320d42b-af84-46aa-b036-812a3ca9c786-kube-api-access-hql4x\") pod \"maas-api-revoke-sub-46494c32-j9bxf\" (UID: \"6320d42b-af84-46aa-b036-812a3ca9c786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf" Nov 03 02:05:40.942494 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.942395 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6320d42b-af84-46aa-b036-812a3ca9c786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46494c32-j9bxf\" (UID: \"6320d42b-af84-46aa-b036-812a3ca9c786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf" Nov 03 02:05:40.942494 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.942441 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6320d42b-af84-46aa-b036-812a3ca9c786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46494c32-j9bxf\" (UID: \"6320d42b-af84-46aa-b036-812a3ca9c786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf" Nov 03 02:05:40.943182 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.943154 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6320d42b-af84-46aa-b036-812a3ca9c786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46494c32-j9bxf\" (UID: \"6320d42b-af84-46aa-b036-812a3ca9c786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf" Nov 03 02:05:40.945274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:40.945252 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6320d42b-af84-46aa-b036-812a3ca9c786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46494c32-j9bxf\" (UID: \"6320d42b-af84-46aa-b036-812a3ca9c786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf" Nov 03 02:05:41.005921 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:41.005838 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hql4x\" (UniqueName: \"kubernetes.io/projected/6320d42b-af84-46aa-b036-812a3ca9c786-kube-api-access-hql4x\") pod \"maas-api-revoke-sub-46494c32-j9bxf\" (UID: \"6320d42b-af84-46aa-b036-812a3ca9c786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf" Nov 03 02:05:41.232181 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:41.232148 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf" Nov 03 02:05:41.608641 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:05:41.608610 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6320d42b_af84_46aa_b036_812a3ca9c786.slice/crio-2f272b04274541ad7a5ad4f5a557c2918b8915d7218eddbc71c6d8aa60e676e1 WatchSource:0}: Error finding container 2f272b04274541ad7a5ad4f5a557c2918b8915d7218eddbc71c6d8aa60e676e1: Status 404 returned error can't find the container with id 2f272b04274541ad7a5ad4f5a557c2918b8915d7218eddbc71c6d8aa60e676e1 Nov 03 02:05:41.610993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:41.610971 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf"] Nov 03 02:05:41.701323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:41.701295 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf" event={"ID":"6320d42b-af84-46aa-b036-812a3ca9c786","Type":"ContainerStarted","Data":"2f272b04274541ad7a5ad4f5a557c2918b8915d7218eddbc71c6d8aa60e676e1"} Nov 03 02:05:42.443353 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:42.443321 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dab070df-285a-4b9f-8d66-b0b6f976cecb" path="/var/lib/kubelet/pods/dab070df-285a-4b9f-8d66-b0b6f976cecb/volumes" Nov 03 02:05:42.709486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:42.709392 5463 generic.go:356] "Generic (PLEG): container finished" podID="6320d42b-af84-46aa-b036-812a3ca9c786" containerID="3d8f8c7c7f7d5f4946386d2350cddd3378e71b32a6e8265ec328f1399f0aa85d" exitCode=0 Nov 03 02:05:42.709486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:42.709461 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf" event={"ID":"6320d42b-af84-46aa-b036-812a3ca9c786","Type":"ContainerDied","Data":"3d8f8c7c7f7d5f4946386d2350cddd3378e71b32a6e8265ec328f1399f0aa85d"} Nov 03 02:05:43.007128 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:05:43.007040 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:05:43.860550 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:43.860527 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf" Nov 03 02:05:43.874212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:43.874171 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6320d42b-af84-46aa-b036-812a3ca9c786-kube-api-access-hql4x\" (UniqueName: \"kubernetes.io/projected/6320d42b-af84-46aa-b036-812a3ca9c786-kube-api-access-hql4x\") pod \"6320d42b-af84-46aa-b036-812a3ca9c786\" (UID: \"6320d42b-af84-46aa-b036-812a3ca9c786\") " Nov 03 02:05:43.874347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:43.874226 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6320d42b-af84-46aa-b036-812a3ca9c786-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6320d42b-af84-46aa-b036-812a3ca9c786-maas-api-xks-ca\") pod \"6320d42b-af84-46aa-b036-812a3ca9c786\" (UID: \"6320d42b-af84-46aa-b036-812a3ca9c786\") " Nov 03 02:05:43.874347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:43.874282 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6320d42b-af84-46aa-b036-812a3ca9c786-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6320d42b-af84-46aa-b036-812a3ca9c786-maas-api-openshift-ca\") pod \"6320d42b-af84-46aa-b036-812a3ca9c786\" (UID: \"6320d42b-af84-46aa-b036-812a3ca9c786\") " Nov 03 02:05:43.874631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:43.874604 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6320d42b-af84-46aa-b036-812a3ca9c786-maas-api-openshift-ca" pod "6320d42b-af84-46aa-b036-812a3ca9c786" (UID: "6320d42b-af84-46aa-b036-812a3ca9c786"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:05:43.876183 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:43.876157 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6320d42b-af84-46aa-b036-812a3ca9c786-maas-api-xks-ca" pod "6320d42b-af84-46aa-b036-812a3ca9c786" (UID: "6320d42b-af84-46aa-b036-812a3ca9c786"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:05:43.876183 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:43.876161 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6320d42b-af84-46aa-b036-812a3ca9c786-kube-api-access-hql4x" pod "6320d42b-af84-46aa-b036-812a3ca9c786" (UID: "6320d42b-af84-46aa-b036-812a3ca9c786"). InnerVolumeSpecName "kube-api-access-hql4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:05:43.975754 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:43.975687 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hql4x\" (UniqueName: \"kubernetes.io/projected/6320d42b-af84-46aa-b036-812a3ca9c786-kube-api-access-hql4x\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:43.975754 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:43.975720 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6320d42b-af84-46aa-b036-812a3ca9c786-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:43.975754 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:43.975736 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6320d42b-af84-46aa-b036-812a3ca9c786-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:44.319843 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:44.319719 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:44.723173 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:44.723144 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf" Nov 03 02:05:44.723173 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:44.723165 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf" event={"ID":"6320d42b-af84-46aa-b036-812a3ca9c786","Type":"ContainerDied","Data":"2f272b04274541ad7a5ad4f5a557c2918b8915d7218eddbc71c6d8aa60e676e1"} Nov 03 02:05:44.723371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:44.723231 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f272b04274541ad7a5ad4f5a557c2918b8915d7218eddbc71c6d8aa60e676e1" Nov 03 02:05:48.840982 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:48.840948 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:48.854588 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:48.854566 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:49.035244 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:49.035210 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:49.762908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:49.762878 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:05:50.606852 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:50.606825 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84"] Nov 03 02:05:50.607317 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:50.607303 5463 cpu_manager.go:414] "containerMap: removing container" podUID="6320d42b-af84-46aa-b036-812a3ca9c786" containerName="revoke-keys" Nov 03 02:05:50.607376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:50.607318 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6320d42b-af84-46aa-b036-812a3ca9c786" containerName="revoke-keys" Nov 03 02:05:50.607430 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:50.607420 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="6320d42b-af84-46aa-b036-812a3ca9c786" containerName="revoke-keys" Nov 03 02:05:50.610768 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:50.610753 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84" Nov 03 02:05:50.644614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:50.644591 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4fd74ad-37f0-42d3-9d34-1f3779b90206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c802d38-2cn84\" (UID: \"b4fd74ad-37f0-42d3-9d34-1f3779b90206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84" Nov 03 02:05:50.644712 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:50.644645 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4fd74ad-37f0-42d3-9d34-1f3779b90206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c802d38-2cn84\" (UID: \"b4fd74ad-37f0-42d3-9d34-1f3779b90206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84" Nov 03 02:05:50.644764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:50.644715 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dzv5\" (UniqueName: \"kubernetes.io/projected/b4fd74ad-37f0-42d3-9d34-1f3779b90206-kube-api-access-7dzv5\") pod \"maas-api-revoke-sub-1c802d38-2cn84\" (UID: \"b4fd74ad-37f0-42d3-9d34-1f3779b90206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84" Nov 03 02:05:50.662801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:50.662779 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84"] Nov 03 02:05:50.663626 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:50.663478 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:05:50.663626 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:50.663589 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:05:50.709863 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:50.709838 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:05:50.710046 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:50.709964 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:05:50.745849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:50.745819 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4fd74ad-37f0-42d3-9d34-1f3779b90206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c802d38-2cn84\" (UID: \"b4fd74ad-37f0-42d3-9d34-1f3779b90206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84" Nov 03 02:05:50.745980 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:50.745901 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4fd74ad-37f0-42d3-9d34-1f3779b90206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c802d38-2cn84\" (UID: \"b4fd74ad-37f0-42d3-9d34-1f3779b90206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84" Nov 03 02:05:50.745980 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:50.745943 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dzv5\" (UniqueName: \"kubernetes.io/projected/b4fd74ad-37f0-42d3-9d34-1f3779b90206-kube-api-access-7dzv5\") pod \"maas-api-revoke-sub-1c802d38-2cn84\" (UID: \"b4fd74ad-37f0-42d3-9d34-1f3779b90206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84" Nov 03 02:05:50.746977 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:50.746951 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4fd74ad-37f0-42d3-9d34-1f3779b90206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c802d38-2cn84\" (UID: \"b4fd74ad-37f0-42d3-9d34-1f3779b90206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84" Nov 03 02:05:50.749237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:50.749214 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4fd74ad-37f0-42d3-9d34-1f3779b90206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c802d38-2cn84\" (UID: \"b4fd74ad-37f0-42d3-9d34-1f3779b90206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84" Nov 03 02:05:50.802888 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:50.802863 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dzv5\" (UniqueName: \"kubernetes.io/projected/b4fd74ad-37f0-42d3-9d34-1f3779b90206-kube-api-access-7dzv5\") pod \"maas-api-revoke-sub-1c802d38-2cn84\" (UID: \"b4fd74ad-37f0-42d3-9d34-1f3779b90206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84" Nov 03 02:05:50.923119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:50.923093 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84" Nov 03 02:05:51.295933 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:05:51.295898 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fd74ad_37f0_42d3_9d34_1f3779b90206.slice/crio-6ea44034c66f3efbe2f2aa77ac0fa17b63827a92d206152812cdfda88ab37d16 WatchSource:0}: Error finding container 6ea44034c66f3efbe2f2aa77ac0fa17b63827a92d206152812cdfda88ab37d16: Status 404 returned error can't find the container with id 6ea44034c66f3efbe2f2aa77ac0fa17b63827a92d206152812cdfda88ab37d16 Nov 03 02:05:51.298474 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:51.298423 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84"] Nov 03 02:05:51.770606 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:51.770571 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84" event={"ID":"b4fd74ad-37f0-42d3-9d34-1f3779b90206","Type":"ContainerStarted","Data":"91208b156a2e1d95fa6bfbbe3a094da993ad1321125ebe51b2502425b55a7405"} Nov 03 02:05:51.770954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:51.770631 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84" event={"ID":"b4fd74ad-37f0-42d3-9d34-1f3779b90206","Type":"ContainerStarted","Data":"6ea44034c66f3efbe2f2aa77ac0fa17b63827a92d206152812cdfda88ab37d16"} Nov 03 02:05:51.886326 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:51.886232 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84" podStartSLOduration=1.886219656 podStartE2EDuration="1.886219656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:05:51.883852812 +0000 UTC m=+833.567961978" watchObservedRunningTime="2026-11-03 02:05:51.886219656 +0000 UTC m=+833.570328817" Nov 03 02:05:51.936279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:51.936247 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj"] Nov 03 02:05:51.940285 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:51.940262 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj" Nov 03 02:05:51.991305 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:51.991274 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj"] Nov 03 02:05:52.058660 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:52.058628 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-792gr\" (UniqueName: \"kubernetes.io/projected/31542120-c0eb-48c7-9cae-7993a532e893-kube-api-access-792gr\") pod \"maas-api-revoke-sub-ecb34462-rp6gj\" (UID: \"31542120-c0eb-48c7-9cae-7993a532e893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj" Nov 03 02:05:52.058797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:52.058785 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31542120-c0eb-48c7-9cae-7993a532e893-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecb34462-rp6gj\" (UID: \"31542120-c0eb-48c7-9cae-7993a532e893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj" Nov 03 02:05:52.058853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:52.058811 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31542120-c0eb-48c7-9cae-7993a532e893-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecb34462-rp6gj\" (UID: \"31542120-c0eb-48c7-9cae-7993a532e893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj" Nov 03 02:05:52.159489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:52.159463 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31542120-c0eb-48c7-9cae-7993a532e893-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecb34462-rp6gj\" (UID: \"31542120-c0eb-48c7-9cae-7993a532e893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj" Nov 03 02:05:52.159621 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:52.159604 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-792gr\" (UniqueName: \"kubernetes.io/projected/31542120-c0eb-48c7-9cae-7993a532e893-kube-api-access-792gr\") pod \"maas-api-revoke-sub-ecb34462-rp6gj\" (UID: \"31542120-c0eb-48c7-9cae-7993a532e893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj" Nov 03 02:05:52.159735 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:52.159722 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31542120-c0eb-48c7-9cae-7993a532e893-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecb34462-rp6gj\" (UID: \"31542120-c0eb-48c7-9cae-7993a532e893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj" Nov 03 02:05:52.160299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:52.160278 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31542120-c0eb-48c7-9cae-7993a532e893-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecb34462-rp6gj\" (UID: \"31542120-c0eb-48c7-9cae-7993a532e893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj" Nov 03 02:05:52.161586 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:52.161568 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31542120-c0eb-48c7-9cae-7993a532e893-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecb34462-rp6gj\" (UID: \"31542120-c0eb-48c7-9cae-7993a532e893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj" Nov 03 02:05:52.217067 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:52.217034 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-792gr\" (UniqueName: \"kubernetes.io/projected/31542120-c0eb-48c7-9cae-7993a532e893-kube-api-access-792gr\") pod \"maas-api-revoke-sub-ecb34462-rp6gj\" (UID: \"31542120-c0eb-48c7-9cae-7993a532e893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj" Nov 03 02:05:52.254351 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:52.254322 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj" Nov 03 02:05:52.623163 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:05:52.623133 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31542120_c0eb_48c7_9cae_7993a532e893.slice/crio-29e309a0ee43c83da84b08154342b80798380f59c908a456eb8351659d7262d8 WatchSource:0}: Error finding container 29e309a0ee43c83da84b08154342b80798380f59c908a456eb8351659d7262d8: Status 404 returned error can't find the container with id 29e309a0ee43c83da84b08154342b80798380f59c908a456eb8351659d7262d8 Nov 03 02:05:52.626047 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:52.626019 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj"] Nov 03 02:05:52.779100 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:52.779071 5463 generic.go:356] "Generic (PLEG): container finished" podID="b4fd74ad-37f0-42d3-9d34-1f3779b90206" containerID="91208b156a2e1d95fa6bfbbe3a094da993ad1321125ebe51b2502425b55a7405" exitCode=0 Nov 03 02:05:52.779474 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:52.779168 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84" event={"ID":"b4fd74ad-37f0-42d3-9d34-1f3779b90206","Type":"ContainerDied","Data":"91208b156a2e1d95fa6bfbbe3a094da993ad1321125ebe51b2502425b55a7405"} Nov 03 02:05:52.780507 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:52.780487 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj" event={"ID":"31542120-c0eb-48c7-9cae-7993a532e893","Type":"ContainerStarted","Data":"29e309a0ee43c83da84b08154342b80798380f59c908a456eb8351659d7262d8"} Nov 03 02:05:53.691620 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:53.691589 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:53.790547 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:53.790511 5463 generic.go:356] "Generic (PLEG): container finished" podID="31542120-c0eb-48c7-9cae-7993a532e893" containerID="e8aaa70c6ee5d24ef939d3a46b7842e79c7300633a420787ffa771c0692a74e2" exitCode=0 Nov 03 02:05:53.790950 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:53.790599 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj" event={"ID":"31542120-c0eb-48c7-9cae-7993a532e893","Type":"ContainerDied","Data":"e8aaa70c6ee5d24ef939d3a46b7842e79c7300633a420787ffa771c0692a74e2"} Nov 03 02:05:53.927953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:53.927930 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84" Nov 03 02:05:53.979868 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:53.979837 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4fd74ad-37f0-42d3-9d34-1f3779b90206-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4fd74ad-37f0-42d3-9d34-1f3779b90206-maas-api-openshift-ca\") pod \"b4fd74ad-37f0-42d3-9d34-1f3779b90206\" (UID: \"b4fd74ad-37f0-42d3-9d34-1f3779b90206\") " Nov 03 02:05:53.980022 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:53.980008 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4fd74ad-37f0-42d3-9d34-1f3779b90206-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4fd74ad-37f0-42d3-9d34-1f3779b90206-maas-api-xks-ca\") pod \"b4fd74ad-37f0-42d3-9d34-1f3779b90206\" (UID: \"b4fd74ad-37f0-42d3-9d34-1f3779b90206\") " Nov 03 02:05:53.980084 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:53.980038 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4fd74ad-37f0-42d3-9d34-1f3779b90206-kube-api-access-7dzv5\" (UniqueName: \"kubernetes.io/projected/b4fd74ad-37f0-42d3-9d34-1f3779b90206-kube-api-access-7dzv5\") pod \"b4fd74ad-37f0-42d3-9d34-1f3779b90206\" (UID: \"b4fd74ad-37f0-42d3-9d34-1f3779b90206\") " Nov 03 02:05:53.980253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:53.980230 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fd74ad-37f0-42d3-9d34-1f3779b90206-maas-api-openshift-ca" pod "b4fd74ad-37f0-42d3-9d34-1f3779b90206" (UID: "b4fd74ad-37f0-42d3-9d34-1f3779b90206"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:05:53.980480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:53.980455 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4fd74ad-37f0-42d3-9d34-1f3779b90206-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:53.982313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:53.982290 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4fd74ad-37f0-42d3-9d34-1f3779b90206-maas-api-xks-ca" pod "b4fd74ad-37f0-42d3-9d34-1f3779b90206" (UID: "b4fd74ad-37f0-42d3-9d34-1f3779b90206"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:05:53.982313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:53.982305 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fd74ad-37f0-42d3-9d34-1f3779b90206-kube-api-access-7dzv5" pod "b4fd74ad-37f0-42d3-9d34-1f3779b90206" (UID: "b4fd74ad-37f0-42d3-9d34-1f3779b90206"). InnerVolumeSpecName "kube-api-access-7dzv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:05:54.082146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:54.082109 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4fd74ad-37f0-42d3-9d34-1f3779b90206-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:54.082146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:54.082148 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dzv5\" (UniqueName: \"kubernetes.io/projected/b4fd74ad-37f0-42d3-9d34-1f3779b90206-kube-api-access-7dzv5\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:54.797676 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:54.797606 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84" Nov 03 02:05:54.797676 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:54.797647 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84" event={"ID":"b4fd74ad-37f0-42d3-9d34-1f3779b90206","Type":"ContainerDied","Data":"6ea44034c66f3efbe2f2aa77ac0fa17b63827a92d206152812cdfda88ab37d16"} Nov 03 02:05:54.798077 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:54.797699 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ea44034c66f3efbe2f2aa77ac0fa17b63827a92d206152812cdfda88ab37d16" Nov 03 02:05:54.948660 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:54.948634 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj" Nov 03 02:05:55.093116 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:55.093022 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31542120-c0eb-48c7-9cae-7993a532e893-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31542120-c0eb-48c7-9cae-7993a532e893-maas-api-openshift-ca\") pod \"31542120-c0eb-48c7-9cae-7993a532e893\" (UID: \"31542120-c0eb-48c7-9cae-7993a532e893\") " Nov 03 02:05:55.093116 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:55.093070 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31542120-c0eb-48c7-9cae-7993a532e893-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31542120-c0eb-48c7-9cae-7993a532e893-maas-api-xks-ca\") pod \"31542120-c0eb-48c7-9cae-7993a532e893\" (UID: \"31542120-c0eb-48c7-9cae-7993a532e893\") " Nov 03 02:05:55.093392 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:55.093155 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31542120-c0eb-48c7-9cae-7993a532e893-kube-api-access-792gr\" (UniqueName: \"kubernetes.io/projected/31542120-c0eb-48c7-9cae-7993a532e893-kube-api-access-792gr\") pod \"31542120-c0eb-48c7-9cae-7993a532e893\" (UID: \"31542120-c0eb-48c7-9cae-7993a532e893\") " Nov 03 02:05:55.093480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:55.093451 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31542120-c0eb-48c7-9cae-7993a532e893-maas-api-openshift-ca" pod "31542120-c0eb-48c7-9cae-7993a532e893" (UID: "31542120-c0eb-48c7-9cae-7993a532e893"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:05:55.093926 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:55.093898 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31542120-c0eb-48c7-9cae-7993a532e893-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:55.097222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:55.096167 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31542120-c0eb-48c7-9cae-7993a532e893-maas-api-xks-ca" pod "31542120-c0eb-48c7-9cae-7993a532e893" (UID: "31542120-c0eb-48c7-9cae-7993a532e893"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:05:55.097222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:55.096314 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31542120-c0eb-48c7-9cae-7993a532e893-kube-api-access-792gr" pod "31542120-c0eb-48c7-9cae-7993a532e893" (UID: "31542120-c0eb-48c7-9cae-7993a532e893"). InnerVolumeSpecName "kube-api-access-792gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:05:55.197216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:55.196519 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31542120-c0eb-48c7-9cae-7993a532e893-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:55.197216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:55.196551 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-792gr\" (UniqueName: \"kubernetes.io/projected/31542120-c0eb-48c7-9cae-7993a532e893-kube-api-access-792gr\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:05:55.811446 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:55.811403 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj" event={"ID":"31542120-c0eb-48c7-9cae-7993a532e893","Type":"ContainerDied","Data":"29e309a0ee43c83da84b08154342b80798380f59c908a456eb8351659d7262d8"} Nov 03 02:05:55.811969 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:55.811473 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29e309a0ee43c83da84b08154342b80798380f59c908a456eb8351659d7262d8" Nov 03 02:05:55.811969 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:55.811549 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj" Nov 03 02:05:58.438795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:05:58.438756 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:02.445208 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:02.445154 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:02.618762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:02.618722 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:03.217665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:03.217606 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:07.904413 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:07.904373 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:09.785808 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:09.785766 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:12.646731 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:12.646696 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:12.806597 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:12.806559 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:13.010836 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:06:13.010753 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:06:13.038636 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:13.038603 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:13.938133 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:13.938100 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8"] Nov 03 02:06:13.938637 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:13.938619 5463 cpu_manager.go:414] "containerMap: removing container" podUID="b4fd74ad-37f0-42d3-9d34-1f3779b90206" containerName="revoke-keys" Nov 03 02:06:13.938637 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:13.938631 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4fd74ad-37f0-42d3-9d34-1f3779b90206" containerName="revoke-keys" Nov 03 02:06:13.938748 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:13.938645 5463 cpu_manager.go:414] "containerMap: removing container" podUID="31542120-c0eb-48c7-9cae-7993a532e893" containerName="revoke-keys" Nov 03 02:06:13.938748 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:13.938650 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31542120-c0eb-48c7-9cae-7993a532e893" containerName="revoke-keys" Nov 03 02:06:13.938748 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:13.938740 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4fd74ad-37f0-42d3-9d34-1f3779b90206" containerName="revoke-keys" Nov 03 02:06:13.938748 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:13.938748 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="31542120-c0eb-48c7-9cae-7993a532e893" containerName="revoke-keys" Nov 03 02:06:13.942092 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:13.942067 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" Nov 03 02:06:14.002233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:13.997214 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8"] Nov 03 02:06:14.020244 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.020207 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-plugins-config-volume\") pod \"payload-processing-external-model-5db9ffd6f8-2hgf8\" (UID: \"0ec59c0c-5e04-438c-84a3-1b4f28d8f609\") " pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" Nov 03 02:06:14.020377 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.020363 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-routing-overlay-volume\") pod \"payload-processing-external-model-5db9ffd6f8-2hgf8\" (UID: \"0ec59c0c-5e04-438c-84a3-1b4f28d8f609\") " pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" Nov 03 02:06:14.020416 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.020389 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-provider-credentials\") pod \"payload-processing-external-model-5db9ffd6f8-2hgf8\" (UID: \"0ec59c0c-5e04-438c-84a3-1b4f28d8f609\") " pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" Nov 03 02:06:14.020416 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.020407 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx89p\" (UniqueName: \"kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-kube-api-access-bx89p\") pod \"payload-processing-external-model-5db9ffd6f8-2hgf8\" (UID: \"0ec59c0c-5e04-438c-84a3-1b4f28d8f609\") " pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" Nov 03 02:06:14.043669 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.043643 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-e895c513fa82\"" Nov 03 02:06:14.043669 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.043661 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1dca5dd705c7\"" Nov 03 02:06:14.043844 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.043752 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-kf7p8\"" Nov 03 02:06:14.043844 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.043769 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 02:06:14.043844 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.043788 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 02:06:14.045728 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.045717 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 02:06:14.046051 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.046034 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 02:06:14.121557 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.121529 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-routing-overlay-volume\") pod \"payload-processing-external-model-5db9ffd6f8-2hgf8\" (UID: \"0ec59c0c-5e04-438c-84a3-1b4f28d8f609\") " pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" Nov 03 02:06:14.121779 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.121562 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-provider-credentials\") pod \"payload-processing-external-model-5db9ffd6f8-2hgf8\" (UID: \"0ec59c0c-5e04-438c-84a3-1b4f28d8f609\") " pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" Nov 03 02:06:14.121779 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.121582 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx89p\" (UniqueName: \"kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-kube-api-access-bx89p\") pod \"payload-processing-external-model-5db9ffd6f8-2hgf8\" (UID: \"0ec59c0c-5e04-438c-84a3-1b4f28d8f609\") " pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" Nov 03 02:06:14.121779 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.121667 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-plugins-config-volume\") pod \"payload-processing-external-model-5db9ffd6f8-2hgf8\" (UID: \"0ec59c0c-5e04-438c-84a3-1b4f28d8f609\") " pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" Nov 03 02:06:14.122242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.122219 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-routing-overlay-volume\") pod \"payload-processing-external-model-5db9ffd6f8-2hgf8\" (UID: \"0ec59c0c-5e04-438c-84a3-1b4f28d8f609\") " pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" Nov 03 02:06:14.122379 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.122360 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-plugins-config-volume\") pod \"payload-processing-external-model-5db9ffd6f8-2hgf8\" (UID: \"0ec59c0c-5e04-438c-84a3-1b4f28d8f609\") " pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" Nov 03 02:06:14.124297 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.124273 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-provider-credentials\") pod \"payload-processing-external-model-5db9ffd6f8-2hgf8\" (UID: \"0ec59c0c-5e04-438c-84a3-1b4f28d8f609\") " pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" Nov 03 02:06:14.178659 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.178633 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx89p\" (UniqueName: \"kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-kube-api-access-bx89p\") pod \"payload-processing-external-model-5db9ffd6f8-2hgf8\" (UID: \"0ec59c0c-5e04-438c-84a3-1b4f28d8f609\") " pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" Nov 03 02:06:14.256399 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.256320 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" Nov 03 02:06:14.624208 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:06:14.624169 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec59c0c_5e04_438c_84a3_1b4f28d8f609.slice/crio-3bd12455c8245bcf195c12e814ce068a5cacb9ff4418b6b84dccc6cc5834b99e WatchSource:0}: Error finding container 3bd12455c8245bcf195c12e814ce068a5cacb9ff4418b6b84dccc6cc5834b99e: Status 404 returned error can't find the container with id 3bd12455c8245bcf195c12e814ce068a5cacb9ff4418b6b84dccc6cc5834b99e Nov 03 02:06:14.627710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.627686 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8"] Nov 03 02:06:14.934977 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.934937 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" event={"ID":"0ec59c0c-5e04-438c-84a3-1b4f28d8f609","Type":"ContainerStarted","Data":"fac100833716d141d3d4dbe26b1bee1f04297680eb1710235b554cf3c4314f72"} Nov 03 02:06:14.935166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.934985 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" event={"ID":"0ec59c0c-5e04-438c-84a3-1b4f28d8f609","Type":"ContainerStarted","Data":"3bd12455c8245bcf195c12e814ce068a5cacb9ff4418b6b84dccc6cc5834b99e"} Nov 03 02:06:14.935166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:14.935017 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" Nov 03 02:06:15.050520 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:15.050460 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" podStartSLOduration=2.050446963 podStartE2EDuration="2.050446963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:15.048604224 +0000 UTC m=+856.732713380" watchObservedRunningTime="2026-11-03 02:06:15.050446963 +0000 UTC m=+856.734556150" Nov 03 02:06:17.418665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:17.418631 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:22.108963 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:22.108921 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:25.944594 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:25.944564 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" Nov 03 02:06:26.857454 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:26.857390 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:30.804338 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:30.804306 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:31.673505 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:31.673469 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:34.774746 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:34.774713 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg"] Nov 03 02:06:34.780098 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:34.780076 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg" Nov 03 02:06:34.832095 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:34.832069 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg"] Nov 03 02:06:34.862043 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:34.861996 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2dtn\" (UniqueName: \"kubernetes.io/projected/29ebdadb-dc6c-4cef-952a-04f78c57e88c-kube-api-access-n2dtn\") pod \"maas-api-revoke-sub-5fb051bd-dn4zg\" (UID: \"29ebdadb-dc6c-4cef-952a-04f78c57e88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg" Nov 03 02:06:34.862169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:34.862143 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29ebdadb-dc6c-4cef-952a-04f78c57e88c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb051bd-dn4zg\" (UID: \"29ebdadb-dc6c-4cef-952a-04f78c57e88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg" Nov 03 02:06:34.862387 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:34.862369 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29ebdadb-dc6c-4cef-952a-04f78c57e88c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb051bd-dn4zg\" (UID: \"29ebdadb-dc6c-4cef-952a-04f78c57e88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg" Nov 03 02:06:34.878939 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:34.878919 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:06:34.879049 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:34.878956 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:06:34.879049 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:34.878964 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:06:34.883057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:34.883038 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:06:34.962959 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:34.962927 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29ebdadb-dc6c-4cef-952a-04f78c57e88c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb051bd-dn4zg\" (UID: \"29ebdadb-dc6c-4cef-952a-04f78c57e88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg" Nov 03 02:06:34.963179 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:34.962986 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2dtn\" (UniqueName: \"kubernetes.io/projected/29ebdadb-dc6c-4cef-952a-04f78c57e88c-kube-api-access-n2dtn\") pod \"maas-api-revoke-sub-5fb051bd-dn4zg\" (UID: \"29ebdadb-dc6c-4cef-952a-04f78c57e88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg" Nov 03 02:06:34.963179 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:34.963045 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29ebdadb-dc6c-4cef-952a-04f78c57e88c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb051bd-dn4zg\" (UID: \"29ebdadb-dc6c-4cef-952a-04f78c57e88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg" Nov 03 02:06:34.964226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:34.964181 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29ebdadb-dc6c-4cef-952a-04f78c57e88c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb051bd-dn4zg\" (UID: \"29ebdadb-dc6c-4cef-952a-04f78c57e88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg" Nov 03 02:06:34.965973 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:34.965949 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29ebdadb-dc6c-4cef-952a-04f78c57e88c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb051bd-dn4zg\" (UID: \"29ebdadb-dc6c-4cef-952a-04f78c57e88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg" Nov 03 02:06:35.020495 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.020467 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2dtn\" (UniqueName: \"kubernetes.io/projected/29ebdadb-dc6c-4cef-952a-04f78c57e88c-kube-api-access-n2dtn\") pod \"maas-api-revoke-sub-5fb051bd-dn4zg\" (UID: \"29ebdadb-dc6c-4cef-952a-04f78c57e88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg" Nov 03 02:06:35.099920 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.099609 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg" Nov 03 02:06:35.224670 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.224637 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds"] Nov 03 02:06:35.235526 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.229873 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.245226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.242924 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd"] Nov 03 02:06:35.255500 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.255473 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.323943 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.323914 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds"] Nov 03 02:06:35.370163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.369080 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.370163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.369134 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnh94\" (UniqueName: \"kubernetes.io/projected/6fc604c8-be34-40a7-9fb8-a07424233eb6-kube-api-access-mnh94\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.370163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.369235 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.370163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.369262 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.370163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.369289 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.370163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.369334 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fc604c8-be34-40a7-9fb8-a07424233eb6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.370163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.369375 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.370163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.369405 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.370163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.369450 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.370163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.369480 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.370163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.369511 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.370163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.369537 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfbpc\" (UniqueName: \"kubernetes.io/projected/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-kube-api-access-nfbpc\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.370163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.369572 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.370163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.369609 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.370163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.369650 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.370163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.369692 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.370163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.369740 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.370163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.369811 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.370905 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.370361 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd"] Nov 03 02:06:35.470435 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.470394 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.470640 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.470442 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnh94\" (UniqueName: \"kubernetes.io/projected/6fc604c8-be34-40a7-9fb8-a07424233eb6-kube-api-access-mnh94\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.470640 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.470504 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.470640 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.470526 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.470640 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.470550 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.470640 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.470593 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fc604c8-be34-40a7-9fb8-a07424233eb6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.470640 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.470633 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.470955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.470662 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.470955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.470705 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.470955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.470735 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.470955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.470759 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.470955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.470947 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.471222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.470784 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfbpc\" (UniqueName: \"kubernetes.io/projected/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-kube-api-access-nfbpc\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.471222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.471044 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.471222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.471114 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.471392 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.471250 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.471503 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.471085 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.471655 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.471638 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.471787 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.471764 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.471867 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.471767 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.471867 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.471839 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fc604c8-be34-40a7-9fb8-a07424233eb6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.471867 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.471853 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.471867 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.471518 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.472077 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.471940 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.472077 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.471945 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.472184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.472090 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.472359 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.472337 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.472661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.472639 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.474126 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.474096 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.474379 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.474357 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.474445 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.474377 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.474445 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.474430 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.489075 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:06:35.489049 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ebdadb_dc6c_4cef_952a_04f78c57e88c.slice/crio-84bfbb45d11a733a3a4a742efd94bd6f411f8f89f59f706209b97bac0899af44 WatchSource:0}: Error finding container 84bfbb45d11a733a3a4a742efd94bd6f411f8f89f59f706209b97bac0899af44: Status 404 returned error can't find the container with id 84bfbb45d11a733a3a4a742efd94bd6f411f8f89f59f706209b97bac0899af44 Nov 03 02:06:35.493261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.493232 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg"] Nov 03 02:06:35.527900 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.527874 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnh94\" (UniqueName: \"kubernetes.io/projected/6fc604c8-be34-40a7-9fb8-a07424233eb6-kube-api-access-mnh94\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.528000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.527871 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.528481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.528456 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jlvds\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.528698 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.528678 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfbpc\" (UniqueName: \"kubernetes.io/projected/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-kube-api-access-nfbpc\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gcscd\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.571365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.571341 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:35.580983 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.580960 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:35.991530 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:06:35.991479 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc604c8_be34_40a7_9fb8_a07424233eb6.slice/crio-6ad69ab03408df4865bdac5a08acd56762c0ceb802f2da9306d117795ea05aba WatchSource:0}: Error finding container 6ad69ab03408df4865bdac5a08acd56762c0ceb802f2da9306d117795ea05aba: Status 404 returned error can't find the container with id 6ad69ab03408df4865bdac5a08acd56762c0ceb802f2da9306d117795ea05aba Nov 03 02:06:35.993351 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.993328 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds"] Nov 03 02:06:35.994775 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:06:35.994753 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be8748c_8d78_4b2e_86ee_d3198bf9cf3b.slice/crio-3b3da3fe43224b17ea867897ae4e402d1eb062b596c441c1d922047ea1f52d22 WatchSource:0}: Error finding container 3b3da3fe43224b17ea867897ae4e402d1eb062b596c441c1d922047ea1f52d22: Status 404 returned error can't find the container with id 3b3da3fe43224b17ea867897ae4e402d1eb062b596c441c1d922047ea1f52d22 Nov 03 02:06:35.995463 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.995428 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:35.995562 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.995489 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:35.995562 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.995522 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:35.997066 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.997038 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd"] Nov 03 02:06:35.997679 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.997644 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:35.997766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.997722 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:35.997832 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:35.997770 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:36.124294 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:36.124253 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg" event={"ID":"29ebdadb-dc6c-4cef-952a-04f78c57e88c","Type":"ContainerStarted","Data":"a8754b43bb01b6d2f3693be891a1973a837f18b95dad122d252bf43ea74a99ad"} Nov 03 02:06:36.124484 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:36.124342 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg" event={"ID":"29ebdadb-dc6c-4cef-952a-04f78c57e88c","Type":"ContainerStarted","Data":"84bfbb45d11a733a3a4a742efd94bd6f411f8f89f59f706209b97bac0899af44"} Nov 03 02:06:36.128070 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:36.127915 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" event={"ID":"6fc604c8-be34-40a7-9fb8-a07424233eb6","Type":"ContainerStarted","Data":"1de919e861c6dac637ee1dfaf21529240cb2745a3f3a14aba1847a455be02bcb"} Nov 03 02:06:36.128070 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:36.127969 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" event={"ID":"6fc604c8-be34-40a7-9fb8-a07424233eb6","Type":"ContainerStarted","Data":"6ad69ab03408df4865bdac5a08acd56762c0ceb802f2da9306d117795ea05aba"} Nov 03 02:06:36.130778 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:36.130751 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" event={"ID":"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b","Type":"ContainerStarted","Data":"f1530ab2c5f087685bd88bb9704620117cd78d4969b531147adfbf0aff897429"} Nov 03 02:06:36.130890 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:36.130799 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" event={"ID":"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b","Type":"ContainerStarted","Data":"3b3da3fe43224b17ea867897ae4e402d1eb062b596c441c1d922047ea1f52d22"} Nov 03 02:06:36.242889 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:36.242788 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg" podStartSLOduration=2.242768783 podStartE2EDuration="2.242768783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:36.237854189 +0000 UTC m=+877.921963344" watchObservedRunningTime="2026-11-03 02:06:36.242768783 +0000 UTC m=+877.926877941" Nov 03 02:06:36.355900 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:36.355836 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" podStartSLOduration=1.355813941 podStartE2EDuration="1.355813941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:36.353451825 +0000 UTC m=+878.037560991" watchObservedRunningTime="2026-11-03 02:06:36.355813941 +0000 UTC m=+878.039923096" Nov 03 02:06:36.470792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:36.470727 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" podStartSLOduration=1.4707096 podStartE2EDuration="1.4707096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:36.469262642 +0000 UTC m=+878.153371790" watchObservedRunningTime="2026-11-03 02:06:36.4707096 +0000 UTC m=+878.154818756" Nov 03 02:06:36.543067 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:36.542971 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:36.572145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:36.572060 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:36.586352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:36.586263 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:37.138929 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:37.138894 5463 generic.go:356] "Generic (PLEG): container finished" podID="29ebdadb-dc6c-4cef-952a-04f78c57e88c" containerID="a8754b43bb01b6d2f3693be891a1973a837f18b95dad122d252bf43ea74a99ad" exitCode=0 Nov 03 02:06:37.139363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:37.138980 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg" event={"ID":"29ebdadb-dc6c-4cef-952a-04f78c57e88c","Type":"ContainerDied","Data":"a8754b43bb01b6d2f3693be891a1973a837f18b95dad122d252bf43ea74a99ad"} Nov 03 02:06:37.580214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:37.579816 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jlvds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 02:06:37.580214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:37.579875 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" podUID="6fc604c8-be34-40a7-9fb8-a07424233eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 02:06:37.588213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:37.585708 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gcscd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 02:06:37.588213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:37.585762 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" podUID="5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 02:06:38.444895 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:38.444874 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg" Nov 03 02:06:38.604510 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:38.604474 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jlvds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 02:06:38.609214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:38.604744 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" podUID="6fc604c8-be34-40a7-9fb8-a07424233eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 02:06:38.613220 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:38.611052 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gcscd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 02:06:38.613395 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:38.613369 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" podUID="5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 02:06:38.625611 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:38.625469 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ebdadb-dc6c-4cef-952a-04f78c57e88c-kube-api-access-n2dtn\" (UniqueName: \"kubernetes.io/projected/29ebdadb-dc6c-4cef-952a-04f78c57e88c-kube-api-access-n2dtn\") pod \"29ebdadb-dc6c-4cef-952a-04f78c57e88c\" (UID: \"29ebdadb-dc6c-4cef-952a-04f78c57e88c\") " Nov 03 02:06:38.625611 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:38.625606 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29ebdadb-dc6c-4cef-952a-04f78c57e88c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29ebdadb-dc6c-4cef-952a-04f78c57e88c-maas-api-xks-ca\") pod \"29ebdadb-dc6c-4cef-952a-04f78c57e88c\" (UID: \"29ebdadb-dc6c-4cef-952a-04f78c57e88c\") " Nov 03 02:06:38.625751 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:38.625692 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29ebdadb-dc6c-4cef-952a-04f78c57e88c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29ebdadb-dc6c-4cef-952a-04f78c57e88c-maas-api-openshift-ca\") pod \"29ebdadb-dc6c-4cef-952a-04f78c57e88c\" (UID: \"29ebdadb-dc6c-4cef-952a-04f78c57e88c\") " Nov 03 02:06:38.626121 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:38.626091 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ebdadb-dc6c-4cef-952a-04f78c57e88c-maas-api-openshift-ca" pod "29ebdadb-dc6c-4cef-952a-04f78c57e88c" (UID: "29ebdadb-dc6c-4cef-952a-04f78c57e88c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:06:38.628225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:38.628185 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ebdadb-dc6c-4cef-952a-04f78c57e88c-kube-api-access-n2dtn" pod "29ebdadb-dc6c-4cef-952a-04f78c57e88c" (UID: "29ebdadb-dc6c-4cef-952a-04f78c57e88c"). InnerVolumeSpecName "kube-api-access-n2dtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:06:38.629102 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:38.629082 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29ebdadb-dc6c-4cef-952a-04f78c57e88c-maas-api-xks-ca" pod "29ebdadb-dc6c-4cef-952a-04f78c57e88c" (UID: "29ebdadb-dc6c-4cef-952a-04f78c57e88c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:06:38.727167 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:38.727093 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29ebdadb-dc6c-4cef-952a-04f78c57e88c-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:06:38.727167 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:38.727121 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29ebdadb-dc6c-4cef-952a-04f78c57e88c-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:06:38.727167 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:38.727134 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2dtn\" (UniqueName: \"kubernetes.io/projected/29ebdadb-dc6c-4cef-952a-04f78c57e88c-kube-api-access-n2dtn\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:06:39.201424 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:39.201387 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg" event={"ID":"29ebdadb-dc6c-4cef-952a-04f78c57e88c","Type":"ContainerDied","Data":"84bfbb45d11a733a3a4a742efd94bd6f411f8f89f59f706209b97bac0899af44"} Nov 03 02:06:39.201600 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:39.201467 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84bfbb45d11a733a3a4a742efd94bd6f411f8f89f59f706209b97bac0899af44" Nov 03 02:06:39.201600 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:39.201400 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg" Nov 03 02:06:39.576271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:39.574458 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jlvds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 02:06:39.576271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:39.574527 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" podUID="6fc604c8-be34-40a7-9fb8-a07424233eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 02:06:39.599215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:39.586517 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gcscd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 02:06:39.599215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:39.586582 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" podUID="5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 02:06:40.574215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:40.573560 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jlvds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 02:06:40.574215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:40.573622 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" podUID="6fc604c8-be34-40a7-9fb8-a07424233eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 02:06:40.597216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:40.586498 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gcscd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 02:06:40.597216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:40.586555 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" podUID="5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 02:06:41.220883 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:41.220856 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:41.574270 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:41.573617 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jlvds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 02:06:41.574270 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:41.573690 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" podUID="6fc604c8-be34-40a7-9fb8-a07424233eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 02:06:41.588214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:41.581575 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gcscd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 02:06:41.588214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:41.581640 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" podUID="5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 02:06:42.572418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:42.572376 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jlvds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 02:06:42.572865 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:42.572447 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" podUID="6fc604c8-be34-40a7-9fb8-a07424233eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 02:06:42.581817 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:42.581789 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gcscd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 02:06:42.581926 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:42.581875 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" podUID="5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 02:06:43.012173 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:06:43.012129 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:06:43.135607 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.135569 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2"] Nov 03 02:06:43.136315 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.136290 5463 cpu_manager.go:414] "containerMap: removing container" podUID="29ebdadb-dc6c-4cef-952a-04f78c57e88c" containerName="revoke-keys" Nov 03 02:06:43.136315 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.136313 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29ebdadb-dc6c-4cef-952a-04f78c57e88c" containerName="revoke-keys" Nov 03 02:06:43.136498 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.136430 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="29ebdadb-dc6c-4cef-952a-04f78c57e88c" containerName="revoke-keys" Nov 03 02:06:43.139921 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.139900 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:43.195270 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.195226 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2"] Nov 03 02:06:43.229331 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.229300 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-dshm\") pod \"e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:43.229503 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.229480 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-home\") pod \"e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:43.229588 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.229525 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-model-cache\") pod \"e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:43.229588 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.229570 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0052a06a-6e50-4aa4-9455-b419b9f36f34-tls-certs\") pod \"e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:43.229697 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.229597 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-tmp-dir\") pod \"e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:43.229880 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.229853 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjnpl\" (UniqueName: \"kubernetes.io/projected/0052a06a-6e50-4aa4-9455-b419b9f36f34-kube-api-access-wjnpl\") pod \"e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:43.240024 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.239995 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f7d9bb\"/\"e2e-conflict-a-7e89b72d-kserve-self-signed-certs\"" Nov 03 02:06:43.260794 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.260757 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:43.331893 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.331783 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-dshm\") pod \"e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:43.332237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.332214 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-home\") pod \"e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:43.332381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.332364 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-model-cache\") pod \"e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:43.332526 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.332510 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0052a06a-6e50-4aa4-9455-b419b9f36f34-tls-certs\") pod \"e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:43.336422 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.336396 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-tmp-dir\") pod \"e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:43.336681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.336660 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjnpl\" (UniqueName: \"kubernetes.io/projected/0052a06a-6e50-4aa4-9455-b419b9f36f34-kube-api-access-wjnpl\") pod \"e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:43.364681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.332923 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-model-cache\") pod \"e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:43.364893 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.332660 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-home\") pod \"e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:43.365004 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.340832 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-tmp-dir\") pod \"e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:43.365102 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.363168 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0052a06a-6e50-4aa4-9455-b419b9f36f34-tls-certs\") pod \"e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:43.365218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.364470 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-dshm\") pod \"e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:43.397173 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.397146 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjnpl\" (UniqueName: \"kubernetes.io/projected/0052a06a-6e50-4aa4-9455-b419b9f36f34-kube-api-access-wjnpl\") pod \"e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:43.452444 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.452403 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:43.586169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.582771 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gcscd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 02:06:43.586169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.582788 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jlvds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 02:06:43.586169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.582821 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" podUID="6fc604c8-be34-40a7-9fb8-a07424233eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 02:06:43.586169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.582821 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" podUID="5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 02:06:43.857822 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:43.857736 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2"] Nov 03 02:06:43.864742 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:06:43.864409 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0052a06a_6e50_4aa4_9455_b419b9f36f34.slice/crio-a372de8aa6e9da7f83949f56e0c2b8a2504b857973595c6d1dd54a6dac555ae6 WatchSource:0}: Error finding container a372de8aa6e9da7f83949f56e0c2b8a2504b857973595c6d1dd54a6dac555ae6: Status 404 returned error can't find the container with id a372de8aa6e9da7f83949f56e0c2b8a2504b857973595c6d1dd54a6dac555ae6 Nov 03 02:06:44.245727 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:44.245611 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" event={"ID":"0052a06a-6e50-4aa4-9455-b419b9f36f34","Type":"ContainerStarted","Data":"9ef5bf755ec5aeeb77dc449d19b6012a84b349ae7fdf5b0f29ec2529437d8337"} Nov 03 02:06:44.245727 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:44.245669 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" event={"ID":"0052a06a-6e50-4aa4-9455-b419b9f36f34","Type":"ContainerStarted","Data":"a372de8aa6e9da7f83949f56e0c2b8a2504b857973595c6d1dd54a6dac555ae6"} Nov 03 02:06:44.365146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:44.364876 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" podStartSLOduration=1.364859198 podStartE2EDuration="1.364859198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:44.362609177 +0000 UTC m=+886.046718333" watchObservedRunningTime="2026-11-03 02:06:44.364859198 +0000 UTC m=+886.048968355" Nov 03 02:06:44.452845 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:44.452807 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:45.113324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:45.113292 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:45.114654 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:45.114624 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:45.116004 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:45.115980 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:06:45.279124 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:45.277226 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:45.279124 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:45.277979 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:45.279431 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:45.279230 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:06:46.036480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:46.036440 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:50.346648 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.346616 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t"] Nov 03 02:06:50.351177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.351159 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.461703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.461664 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t"] Nov 03 02:06:50.504429 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.504398 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.504566 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.504439 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.504566 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.504462 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.504650 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.504588 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.504650 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.504624 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8gpd\" (UniqueName: \"kubernetes.io/projected/0e38d821-9e7d-46a4-93a1-4b4285b1de35-kube-api-access-s8gpd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.504718 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.504704 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.504752 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.504731 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.504782 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.504765 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.504894 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.504880 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.606072 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.605985 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.606072 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.606030 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.606331 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.606123 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.606331 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.606157 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.606331 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.606218 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8gpd\" (UniqueName: \"kubernetes.io/projected/0e38d821-9e7d-46a4-93a1-4b4285b1de35-kube-api-access-s8gpd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.606331 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.606280 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.606331 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.606315 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.606601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.606354 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.606601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.606480 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.606601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.606502 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.606762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.606666 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.606902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.606876 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.606977 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.606904 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.606977 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.606935 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.608597 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.608573 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.608693 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.608662 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.664881 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.664856 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.665270 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.665253 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8gpd\" (UniqueName: \"kubernetes.io/projected/0e38d821-9e7d-46a4-93a1-4b4285b1de35-kube-api-access-s8gpd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qxq6t\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.666020 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.665989 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:50.832431 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:50.832401 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:51.081221 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:06:51.081166 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e38d821_9e7d_46a4_93a1_4b4285b1de35.slice/crio-a4d95db0617707dd54efa40df0f3f5051461383756d0c006adf87b79180083f7 WatchSource:0}: Error finding container a4d95db0617707dd54efa40df0f3f5051461383756d0c006adf87b79180083f7: Status 404 returned error can't find the container with id a4d95db0617707dd54efa40df0f3f5051461383756d0c006adf87b79180083f7 Nov 03 02:06:51.083597 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:51.083563 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:51.083679 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:51.083634 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:51.083679 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:51.083666 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:51.084801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:51.084780 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t"] Nov 03 02:06:51.316346 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:51.316311 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" event={"ID":"0e38d821-9e7d-46a4-93a1-4b4285b1de35","Type":"ContainerStarted","Data":"ff87290590ce5259684c3b270c0b65b134caaf3c222d9c5bd23b5a982ea5d645"} Nov 03 02:06:51.316552 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:51.316369 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" event={"ID":"0e38d821-9e7d-46a4-93a1-4b4285b1de35","Type":"ContainerStarted","Data":"a4d95db0617707dd54efa40df0f3f5051461383756d0c006adf87b79180083f7"} Nov 03 02:06:51.435551 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:51.435492 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" podStartSLOduration=1.435473321 podStartE2EDuration="1.435473321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:51.432140687 +0000 UTC m=+893.116249841" watchObservedRunningTime="2026-11-03 02:06:51.435473321 +0000 UTC m=+893.119582477" Nov 03 02:06:51.666326 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:51.666283 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:52.677214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:52.672433 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qxq6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 02:06:52.677214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:52.672498 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" podUID="0e38d821-9e7d-46a4-93a1-4b4285b1de35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 02:06:53.453556 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:53.453525 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:53.487424 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:53.487398 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:53.666591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:53.666554 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qxq6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 02:06:53.666882 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:53.666854 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" podUID="0e38d821-9e7d-46a4-93a1-4b4285b1de35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 02:06:54.382122 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:54.382087 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:06:54.672625 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:54.672587 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qxq6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 02:06:54.672826 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:54.672665 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" podUID="0e38d821-9e7d-46a4-93a1-4b4285b1de35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 02:06:55.667214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:55.666579 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qxq6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 02:06:55.667214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:55.666637 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" podUID="0e38d821-9e7d-46a4-93a1-4b4285b1de35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 02:06:55.673837 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:55.673806 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:56.666607 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:56.666570 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qxq6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 02:06:56.666877 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:56.666848 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" podUID="0e38d821-9e7d-46a4-93a1-4b4285b1de35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 02:06:57.667112 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:57.667079 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qxq6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 02:06:57.667622 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:57.667140 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" podUID="0e38d821-9e7d-46a4-93a1-4b4285b1de35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 02:06:58.601157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:58.601131 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:06:58.605044 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:58.605017 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:06:58.667330 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:58.667296 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qxq6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 02:06:58.667706 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:58.667359 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" podUID="0e38d821-9e7d-46a4-93a1-4b4285b1de35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 02:06:59.921518 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:59.921485 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:59.922030 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:59.922010 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:06:59.922865 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:06:59.922847 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:07:00.489959 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:00.489864 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:05.278110 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:05.278072 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:08.344063 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:08.344027 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:09.923338 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:09.923241 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:13.007495 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:07:13.007453 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:07:14.635270 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:14.635238 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:19.333069 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:19.333027 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:21.098798 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:21.098753 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk"] Nov 03 02:07:21.103346 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:21.103317 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" Nov 03 02:07:21.187398 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:21.187362 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3cea42-0b52-4205-a2c7-1d8844030513-plugins-config-volume\") pod \"payload-processing-external-model-6b5d8ddb76-hlngk\" (UID: \"1d3cea42-0b52-4205-a2c7-1d8844030513\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" Nov 03 02:07:21.187570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:21.187454 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d3cea42-0b52-4205-a2c7-1d8844030513-routing-overlay-volume\") pod \"payload-processing-external-model-6b5d8ddb76-hlngk\" (UID: \"1d3cea42-0b52-4205-a2c7-1d8844030513\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" Nov 03 02:07:21.187570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:21.187535 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d3cea42-0b52-4205-a2c7-1d8844030513-provider-credentials\") pod \"payload-processing-external-model-6b5d8ddb76-hlngk\" (UID: \"1d3cea42-0b52-4205-a2c7-1d8844030513\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" Nov 03 02:07:21.187664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:21.187638 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5mtq\" (UniqueName: \"kubernetes.io/projected/1d3cea42-0b52-4205-a2c7-1d8844030513-kube-api-access-c5mtq\") pod \"payload-processing-external-model-6b5d8ddb76-hlngk\" (UID: \"1d3cea42-0b52-4205-a2c7-1d8844030513\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" Nov 03 02:07:21.208898 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:21.208866 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk"] Nov 03 02:07:21.288352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:21.288325 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3cea42-0b52-4205-a2c7-1d8844030513-plugins-config-volume\") pod \"payload-processing-external-model-6b5d8ddb76-hlngk\" (UID: \"1d3cea42-0b52-4205-a2c7-1d8844030513\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" Nov 03 02:07:21.288493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:21.288367 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d3cea42-0b52-4205-a2c7-1d8844030513-routing-overlay-volume\") pod \"payload-processing-external-model-6b5d8ddb76-hlngk\" (UID: \"1d3cea42-0b52-4205-a2c7-1d8844030513\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" Nov 03 02:07:21.288493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:21.288410 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d3cea42-0b52-4205-a2c7-1d8844030513-provider-credentials\") pod \"payload-processing-external-model-6b5d8ddb76-hlngk\" (UID: \"1d3cea42-0b52-4205-a2c7-1d8844030513\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" Nov 03 02:07:21.288613 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:21.288521 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5mtq\" (UniqueName: \"kubernetes.io/projected/1d3cea42-0b52-4205-a2c7-1d8844030513-kube-api-access-c5mtq\") pod \"payload-processing-external-model-6b5d8ddb76-hlngk\" (UID: \"1d3cea42-0b52-4205-a2c7-1d8844030513\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" Nov 03 02:07:21.289130 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:21.289103 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d3cea42-0b52-4205-a2c7-1d8844030513-routing-overlay-volume\") pod \"payload-processing-external-model-6b5d8ddb76-hlngk\" (UID: \"1d3cea42-0b52-4205-a2c7-1d8844030513\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" Nov 03 02:07:21.289290 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:21.289182 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3cea42-0b52-4205-a2c7-1d8844030513-plugins-config-volume\") pod \"payload-processing-external-model-6b5d8ddb76-hlngk\" (UID: \"1d3cea42-0b52-4205-a2c7-1d8844030513\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" Nov 03 02:07:21.290827 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:21.290807 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d3cea42-0b52-4205-a2c7-1d8844030513-provider-credentials\") pod \"payload-processing-external-model-6b5d8ddb76-hlngk\" (UID: \"1d3cea42-0b52-4205-a2c7-1d8844030513\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" Nov 03 02:07:21.347316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:21.347293 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5mtq\" (UniqueName: \"kubernetes.io/projected/1d3cea42-0b52-4205-a2c7-1d8844030513-kube-api-access-c5mtq\") pod \"payload-processing-external-model-6b5d8ddb76-hlngk\" (UID: \"1d3cea42-0b52-4205-a2c7-1d8844030513\") " pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" Nov 03 02:07:21.418347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:21.418316 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" Nov 03 02:07:21.786332 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:07:21.786296 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d3cea42_0b52_4205_a2c7_1d8844030513.slice/crio-6f814b32a18ffdad061448392223dd3d6eb60ead5704ffc17c2815db3d4a6db0 WatchSource:0}: Error finding container 6f814b32a18ffdad061448392223dd3d6eb60ead5704ffc17c2815db3d4a6db0: Status 404 returned error can't find the container with id 6f814b32a18ffdad061448392223dd3d6eb60ead5704ffc17c2815db3d4a6db0 Nov 03 02:07:21.787865 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:21.787841 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk"] Nov 03 02:07:22.575668 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:22.575629 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" event={"ID":"1d3cea42-0b52-4205-a2c7-1d8844030513","Type":"ContainerStarted","Data":"6f6aa69002e204013f0cfe465d5f01f6a0093ca204e911fd9e9f39ccce9c0636"} Nov 03 02:07:22.576178 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:22.575695 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" event={"ID":"1d3cea42-0b52-4205-a2c7-1d8844030513","Type":"ContainerStarted","Data":"6f814b32a18ffdad061448392223dd3d6eb60ead5704ffc17c2815db3d4a6db0"} Nov 03 02:07:22.576178 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:22.575856 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" Nov 03 02:07:22.690409 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:22.690349 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" podStartSLOduration=1.690334033 podStartE2EDuration="1.690334033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:07:22.689501709 +0000 UTC m=+924.373610865" watchObservedRunningTime="2026-11-03 02:07:22.690334033 +0000 UTC m=+924.374443453" Nov 03 02:07:24.042062 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:24.042032 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:24.183672 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:24.183588 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:29.033213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:29.026376 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:29.759216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:29.758283 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 03 02:07:33.585456 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:33.585415 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" podUID="1d3cea42-0b52-4205-a2c7-1d8844030513" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:07:33.717828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:33.717794 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:38.464908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:38.462855 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:39.785528 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:39.785491 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:43.008445 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:07:43.008411 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:07:43.116424 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:43.116386 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:43.583617 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:43.583580 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" podUID="1d3cea42-0b52-4205-a2c7-1d8844030513" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:07:43.914557 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:43.914521 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:44.070440 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:44.070411 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:44.237274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:44.236657 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:47.840534 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:47.840506 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:48.389515 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:48.389478 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk"] Nov 03 02:07:48.390815 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:48.390778 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" podUID="1d3cea42-0b52-4205-a2c7-1d8844030513" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:07:52.537821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:52.537783 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:53.389849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.389796 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" podUID="1d3cea42-0b52-4205-a2c7-1d8844030513" containerName="payload-processing" containerID="cri-o://6f6aa69002e204013f0cfe465d5f01f6a0093ca204e911fd9e9f39ccce9c0636" gracePeriod=25 Nov 03 02:07:53.657377 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.657354 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" Nov 03 02:07:53.777988 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.777952 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d3cea42-0b52-4205-a2c7-1d8844030513-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3cea42-0b52-4205-a2c7-1d8844030513-plugins-config-volume\") pod \"1d3cea42-0b52-4205-a2c7-1d8844030513\" (UID: \"1d3cea42-0b52-4205-a2c7-1d8844030513\") " Nov 03 02:07:53.778157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.778064 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d3cea42-0b52-4205-a2c7-1d8844030513-kube-api-access-c5mtq\" (UniqueName: \"kubernetes.io/projected/1d3cea42-0b52-4205-a2c7-1d8844030513-kube-api-access-c5mtq\") pod \"1d3cea42-0b52-4205-a2c7-1d8844030513\" (UID: \"1d3cea42-0b52-4205-a2c7-1d8844030513\") " Nov 03 02:07:53.778157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.778123 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d3cea42-0b52-4205-a2c7-1d8844030513-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d3cea42-0b52-4205-a2c7-1d8844030513-provider-credentials\") pod \"1d3cea42-0b52-4205-a2c7-1d8844030513\" (UID: \"1d3cea42-0b52-4205-a2c7-1d8844030513\") " Nov 03 02:07:53.778296 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.778225 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d3cea42-0b52-4205-a2c7-1d8844030513-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d3cea42-0b52-4205-a2c7-1d8844030513-routing-overlay-volume\") pod \"1d3cea42-0b52-4205-a2c7-1d8844030513\" (UID: \"1d3cea42-0b52-4205-a2c7-1d8844030513\") " Nov 03 02:07:53.778471 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.778434 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d3cea42-0b52-4205-a2c7-1d8844030513-plugins-config-volume" pod "1d3cea42-0b52-4205-a2c7-1d8844030513" (UID: "1d3cea42-0b52-4205-a2c7-1d8844030513"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:07:53.778725 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.778705 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3cea42-0b52-4205-a2c7-1d8844030513-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:07:53.778889 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.778867 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d3cea42-0b52-4205-a2c7-1d8844030513-routing-overlay-volume" pod "1d3cea42-0b52-4205-a2c7-1d8844030513" (UID: "1d3cea42-0b52-4205-a2c7-1d8844030513"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:07:53.780322 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.780301 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d3cea42-0b52-4205-a2c7-1d8844030513-provider-credentials" pod "1d3cea42-0b52-4205-a2c7-1d8844030513" (UID: "1d3cea42-0b52-4205-a2c7-1d8844030513"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:07:53.780414 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.780323 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d3cea42-0b52-4205-a2c7-1d8844030513-kube-api-access-c5mtq" pod "1d3cea42-0b52-4205-a2c7-1d8844030513" (UID: "1d3cea42-0b52-4205-a2c7-1d8844030513"). InnerVolumeSpecName "kube-api-access-c5mtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:07:53.834721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.834696 5463 generic.go:356] "Generic (PLEG): container finished" podID="1d3cea42-0b52-4205-a2c7-1d8844030513" containerID="6f6aa69002e204013f0cfe465d5f01f6a0093ca204e911fd9e9f39ccce9c0636" exitCode=0 Nov 03 02:07:53.834821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.834744 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" Nov 03 02:07:53.834821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.834784 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" event={"ID":"1d3cea42-0b52-4205-a2c7-1d8844030513","Type":"ContainerDied","Data":"6f6aa69002e204013f0cfe465d5f01f6a0093ca204e911fd9e9f39ccce9c0636"} Nov 03 02:07:53.834903 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.834837 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk" event={"ID":"1d3cea42-0b52-4205-a2c7-1d8844030513","Type":"ContainerDied","Data":"6f814b32a18ffdad061448392223dd3d6eb60ead5704ffc17c2815db3d4a6db0"} Nov 03 02:07:53.834903 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.834859 5463 scope.go:122] "RemoveContainer" containerID="6f6aa69002e204013f0cfe465d5f01f6a0093ca204e911fd9e9f39ccce9c0636" Nov 03 02:07:53.844646 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.844629 5463 scope.go:122] "RemoveContainer" containerID="6f6aa69002e204013f0cfe465d5f01f6a0093ca204e911fd9e9f39ccce9c0636" Nov 03 02:07:53.844915 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:07:53.844898 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f6aa69002e204013f0cfe465d5f01f6a0093ca204e911fd9e9f39ccce9c0636\": container with ID starting with 6f6aa69002e204013f0cfe465d5f01f6a0093ca204e911fd9e9f39ccce9c0636 not found: ID does not exist" containerID="6f6aa69002e204013f0cfe465d5f01f6a0093ca204e911fd9e9f39ccce9c0636" Nov 03 02:07:53.844970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.844922 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f6aa69002e204013f0cfe465d5f01f6a0093ca204e911fd9e9f39ccce9c0636"} err="failed to get container status \"6f6aa69002e204013f0cfe465d5f01f6a0093ca204e911fd9e9f39ccce9c0636\": rpc error: code = NotFound desc = could not find container \"6f6aa69002e204013f0cfe465d5f01f6a0093ca204e911fd9e9f39ccce9c0636\": container with ID starting with 6f6aa69002e204013f0cfe465d5f01f6a0093ca204e911fd9e9f39ccce9c0636 not found: ID does not exist" Nov 03 02:07:53.879643 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.879615 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5mtq\" (UniqueName: \"kubernetes.io/projected/1d3cea42-0b52-4205-a2c7-1d8844030513-kube-api-access-c5mtq\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:07:53.879643 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.879641 5463 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d3cea42-0b52-4205-a2c7-1d8844030513-provider-credentials\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:07:53.879885 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:53.879653 5463 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d3cea42-0b52-4205-a2c7-1d8844030513-routing-overlay-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:07:54.002344 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:54.002276 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk"] Nov 03 02:07:54.006420 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:54.006397 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b5d8ddb76-hlngk"] Nov 03 02:07:54.443316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:54.443287 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d3cea42-0b52-4205-a2c7-1d8844030513" path="/var/lib/kubelet/pods/1d3cea42-0b52-4205-a2c7-1d8844030513/volumes" Nov 03 02:07:55.496708 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:07:55.496675 5463 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1dca5dd705c7: secret "agc-cred-1dca5dd705c7" not found Nov 03 02:07:55.496708 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:07:55.496706 5463 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8: secret "agc-cred-1dca5dd705c7" not found Nov 03 02:07:55.497264 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:07:55.496793 5463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-provider-credentials podName:0ec59c0c-5e04-438c-84a3-1b4f28d8f609 nodeName:}" failed. No retries permitted until 2026-11-03 02:07:55.996771247 +0000 UTC m=+957.680880381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-provider-credentials") pod "payload-processing-external-model-5db9ffd6f8-2hgf8" (UID: "0ec59c0c-5e04-438c-84a3-1b4f28d8f609") : secret "agc-cred-1dca5dd705c7" not found Nov 03 02:07:56.001461 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:07:56.001426 5463 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1dca5dd705c7: secret "agc-cred-1dca5dd705c7" not found Nov 03 02:07:56.001461 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:07:56.001454 5463 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8: secret "agc-cred-1dca5dd705c7" not found Nov 03 02:07:56.001686 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:07:56.001505 5463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-provider-credentials podName:0ec59c0c-5e04-438c-84a3-1b4f28d8f609 nodeName:}" failed. No retries permitted until 2026-11-03 02:07:57.001489669 +0000 UTC m=+958.685598802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-provider-credentials") pod "payload-processing-external-model-5db9ffd6f8-2hgf8" (UID: "0ec59c0c-5e04-438c-84a3-1b4f28d8f609") : secret "agc-cred-1dca5dd705c7" not found Nov 03 02:07:57.011299 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:07:57.011266 5463 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1dca5dd705c7: secret "agc-cred-1dca5dd705c7" not found Nov 03 02:07:57.011299 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:07:57.011295 5463 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8: secret "agc-cred-1dca5dd705c7" not found Nov 03 02:07:57.011743 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:07:57.011377 5463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-provider-credentials podName:0ec59c0c-5e04-438c-84a3-1b4f28d8f609 nodeName:}" failed. No retries permitted until 2026-11-03 02:07:59.011360278 +0000 UTC m=+960.695469427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-provider-credentials") pod "payload-processing-external-model-5db9ffd6f8-2hgf8" (UID: "0ec59c0c-5e04-438c-84a3-1b4f28d8f609") : secret "agc-cred-1dca5dd705c7" not found Nov 03 02:07:57.429213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:57.429165 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:58.014754 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.014724 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k"] Nov 03 02:07:58.015285 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.015270 5463 cpu_manager.go:414] "containerMap: removing container" podUID="1d3cea42-0b52-4205-a2c7-1d8844030513" containerName="payload-processing" Nov 03 02:07:58.015338 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.015286 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d3cea42-0b52-4205-a2c7-1d8844030513" containerName="payload-processing" Nov 03 02:07:58.015398 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.015390 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d3cea42-0b52-4205-a2c7-1d8844030513" containerName="payload-processing" Nov 03 02:07:58.020099 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.020083 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:07:58.072623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.072591 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k"] Nov 03 02:07:58.101198 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.101171 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-521f1f\"/\"openshift-service-ca.crt\"" Nov 03 02:07:58.118390 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.118364 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-521f1f\"/\"kube-root-ca.crt\"" Nov 03 02:07:58.118390 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.118369 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-521f1f\"/\"distinct-w2-521f1f-kserve-self-signed-certs\"" Nov 03 02:07:58.123699 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.123682 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-521f1f\"/\"default-dockercfg-dkd2g\"" Nov 03 02:07:58.125229 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.125183 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8gm2\" (UniqueName: \"kubernetes.io/projected/dc1cc20f-fb60-4334-ba14-5f21203ee975-kube-api-access-r8gm2\") pod \"distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:07:58.125294 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.125247 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1cc20f-fb60-4334-ba14-5f21203ee975-tls-certs\") pod \"distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:07:58.125374 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.125357 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-home\") pod \"distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:07:58.125415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.125387 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-model-cache\") pod \"distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:07:58.125590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.125566 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-dshm\") pod \"distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:07:58.125709 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.125620 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-tmp-dir\") pod \"distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:07:58.126257 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.126236 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:58.226752 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.226713 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8gm2\" (UniqueName: \"kubernetes.io/projected/dc1cc20f-fb60-4334-ba14-5f21203ee975-kube-api-access-r8gm2\") pod \"distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:07:58.226947 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.226757 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1cc20f-fb60-4334-ba14-5f21203ee975-tls-certs\") pod \"distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:07:58.226947 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.226794 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-home\") pod \"distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:07:58.226947 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.226820 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-model-cache\") pod \"distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:07:58.227116 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.226962 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-dshm\") pod \"distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:07:58.227116 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.227030 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-tmp-dir\") pod \"distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:07:58.227294 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.227273 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-home\") pod \"distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:07:58.227351 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.227332 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-model-cache\") pod \"distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:07:58.227429 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.227411 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-tmp-dir\") pod \"distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:07:58.229169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.229145 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-dshm\") pod \"distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:07:58.229469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.229449 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1cc20f-fb60-4334-ba14-5f21203ee975-tls-certs\") pod \"distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:07:58.285367 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.285307 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8gm2\" (UniqueName: \"kubernetes.io/projected/dc1cc20f-fb60-4334-ba14-5f21203ee975-kube-api-access-r8gm2\") pod \"distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:07:58.333515 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.333492 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:07:58.650364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.650334 5463 scope.go:122] "RemoveContainer" containerID="a3d3f9707c98d7bcb9c1c4961535d71422c391bc1728ea80bba9a6466569dc55" Nov 03 02:07:58.659781 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.659753 5463 scope.go:122] "RemoveContainer" containerID="3d325b4a4f9bae73e7768bd0e3d10a22fc244bccd145ae472d59873e89e1dd9d" Nov 03 02:07:58.699333 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:07:58.699309 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc1cc20f_fb60_4334_ba14_5f21203ee975.slice/crio-cbe1958a9ceac1e4d122feba58dca9dbf8549227a2752910d721d30a8b1a1d49 WatchSource:0}: Error finding container cbe1958a9ceac1e4d122feba58dca9dbf8549227a2752910d721d30a8b1a1d49: Status 404 returned error can't find the container with id cbe1958a9ceac1e4d122feba58dca9dbf8549227a2752910d721d30a8b1a1d49 Nov 03 02:07:58.702513 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.702490 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k"] Nov 03 02:07:58.879807 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.879761 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" event={"ID":"dc1cc20f-fb60-4334-ba14-5f21203ee975","Type":"ContainerStarted","Data":"8304743d475d541f0121f8b2e7f18e9f228f904cbf9363239a1792deeee2383a"} Nov 03 02:07:58.879972 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.879828 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" event={"ID":"dc1cc20f-fb60-4334-ba14-5f21203ee975","Type":"ContainerStarted","Data":"cbe1958a9ceac1e4d122feba58dca9dbf8549227a2752910d721d30a8b1a1d49"} Nov 03 02:07:58.996785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:58.996693 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podStartSLOduration=1.996678808 podStartE2EDuration="1.996678808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:07:58.994127521 +0000 UTC m=+960.678236676" watchObservedRunningTime="2026-11-03 02:07:58.996678808 +0000 UTC m=+960.680787964" Nov 03 02:07:59.035897 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:07:59.035870 5463 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1dca5dd705c7: secret "agc-cred-1dca5dd705c7" not found Nov 03 02:07:59.035897 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:07:59.035903 5463 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8: secret "agc-cred-1dca5dd705c7" not found Nov 03 02:07:59.036335 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:07:59.035964 5463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-provider-credentials podName:0ec59c0c-5e04-438c-84a3-1b4f28d8f609 nodeName:}" failed. No retries permitted until 2026-11-03 02:08:03.035945172 +0000 UTC m=+964.720054327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-provider-credentials") pod "payload-processing-external-model-5db9ffd6f8-2hgf8" (UID: "0ec59c0c-5e04-438c-84a3-1b4f28d8f609") : secret "agc-cred-1dca5dd705c7" not found Nov 03 02:07:59.333833 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:07:59.333754 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:08:00.622567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.622529 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq"] Nov 03 02:08:00.629258 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.629241 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.654964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.654559 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.654964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.654632 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.654964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.654744 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.654964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.654806 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ef5394-19c0-4b33-84f6-2087dede03a0-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.654964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.654875 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.654964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.654901 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.654964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.654937 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.654964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.654969 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqvs9\" (UniqueName: \"kubernetes.io/projected/32ef5394-19c0-4b33-84f6-2087dede03a0-kube-api-access-lqvs9\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.655501 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.655009 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.737876 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.737840 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq"] Nov 03 02:08:00.755785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.755747 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.755785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.755783 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.756003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.755814 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.756003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.755837 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqvs9\" (UniqueName: \"kubernetes.io/projected/32ef5394-19c0-4b33-84f6-2087dede03a0-kube-api-access-lqvs9\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.756003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.755863 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.756003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.755965 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.756003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.755998 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.756266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.756023 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.756266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.756058 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ef5394-19c0-4b33-84f6-2087dede03a0-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.756716 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.756685 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.756888 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.756818 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ef5394-19c0-4b33-84f6-2087dede03a0-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.756993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.756943 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.757342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.757319 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.757674 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.757654 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.762691 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.762667 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.766432 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.766407 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.812949 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.812915 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.813065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.812994 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqvs9\" (UniqueName: \"kubernetes.io/projected/32ef5394-19c0-4b33-84f6-2087dede03a0-kube-api-access-lqvs9\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:00.880285 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.880205 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8"] Nov 03 02:08:00.944825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:00.944794 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:01.374681 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:08:01.374636 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ef5394_19c0_4b33_84f6_2087dede03a0.slice/crio-0d14eb679cb15cfdcfb36634c6db2c3fd9a00320ab2e289d43c697775542d5db WatchSource:0}: Error finding container 0d14eb679cb15cfdcfb36634c6db2c3fd9a00320ab2e289d43c697775542d5db: Status 404 returned error can't find the container with id 0d14eb679cb15cfdcfb36634c6db2c3fd9a00320ab2e289d43c697775542d5db Nov 03 02:08:01.377572 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.377533 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:08:01.377670 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.377626 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:08:01.377712 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.377668 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:08:01.377712 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.377690 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq"] Nov 03 02:08:01.769039 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.769007 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58"] Nov 03 02:08:01.773464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.773444 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.855013 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.854685 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58"] Nov 03 02:08:01.872320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.871608 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.872320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.871734 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.872320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.871772 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.872320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.871890 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.872320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.871950 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.872320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.871979 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4lkx\" (UniqueName: \"kubernetes.io/projected/1e5aa4d8-7e07-4efe-96ef-93b35748423e-kube-api-access-t4lkx\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.872320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.872044 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.872320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.872067 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.872320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.872123 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.903998 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.903961 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" event={"ID":"32ef5394-19c0-4b33-84f6-2087dede03a0","Type":"ContainerStarted","Data":"0cfba01290b039c4902ed6b049e39f9411f18fc8eb7fa24bb60395651e1e8792"} Nov 03 02:08:01.904133 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.904021 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" event={"ID":"32ef5394-19c0-4b33-84f6-2087dede03a0","Type":"ContainerStarted","Data":"0d14eb679cb15cfdcfb36634c6db2c3fd9a00320ab2e289d43c697775542d5db"} Nov 03 02:08:01.945329 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.945305 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:01.973002 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.972960 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.973769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.973680 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4lkx\" (UniqueName: \"kubernetes.io/projected/1e5aa4d8-7e07-4efe-96ef-93b35748423e-kube-api-access-t4lkx\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.973906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.973775 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.973906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.973804 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.973906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.973890 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.974086 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.973978 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.974146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.974090 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.974146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.974113 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.974146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.974124 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.974337 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.974291 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.978235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.974634 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.978235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.974804 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.978235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.975052 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.978235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.975342 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.978235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.976827 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:01.978235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:01.977413 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:02.020227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:02.020139 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" podStartSLOduration=2.020128218 podStartE2EDuration="2.020128218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:08:02.019462133 +0000 UTC m=+963.703571288" watchObservedRunningTime="2026-11-03 02:08:02.020128218 +0000 UTC m=+963.704237373" Nov 03 02:08:02.033023 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:02.032992 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:02.033179 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:02.033160 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4lkx\" (UniqueName: \"kubernetes.io/projected/1e5aa4d8-7e07-4efe-96ef-93b35748423e-kube-api-access-t4lkx\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:02.042908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:02.042881 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:02.090643 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:02.090605 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:02.521700 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:08:02.521657 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5aa4d8_7e07_4efe_96ef_93b35748423e.slice/crio-1841c2b3f15a90f77c9968bc2161869fad99127a7b67fd599ed07fdaea54e08c WatchSource:0}: Error finding container 1841c2b3f15a90f77c9968bc2161869fad99127a7b67fd599ed07fdaea54e08c: Status 404 returned error can't find the container with id 1841c2b3f15a90f77c9968bc2161869fad99127a7b67fd599ed07fdaea54e08c Nov 03 02:08:02.524023 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:02.523988 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:08:02.524139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:02.524060 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:08:02.524139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:02.524088 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:08:02.524939 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:02.524918 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58"] Nov 03 02:08:02.924842 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:02.924255 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" event={"ID":"1e5aa4d8-7e07-4efe-96ef-93b35748423e","Type":"ContainerStarted","Data":"c96fe2e59e8dc0de2487dfcd40f84d03fbe88a4677750e2de31b404e8d4c99f0"} Nov 03 02:08:02.925469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:02.925443 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" event={"ID":"1e5aa4d8-7e07-4efe-96ef-93b35748423e","Type":"ContainerStarted","Data":"1841c2b3f15a90f77c9968bc2161869fad99127a7b67fd599ed07fdaea54e08c"} Nov 03 02:08:02.958212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:02.952298 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 02:08:02.958212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:02.952369 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" podUID="32ef5394-19c0-4b33-84f6-2087dede03a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 02:08:03.096213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:03.095595 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:03.097337 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:03.097303 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 02:08:03.097696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:03.097363 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" podUID="1e5aa4d8-7e07-4efe-96ef-93b35748423e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 02:08:03.101172 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:08:03.101147 5463 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1dca5dd705c7: secret "agc-cred-1dca5dd705c7" not found Nov 03 02:08:03.101312 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:08:03.101178 5463 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8: secret "agc-cred-1dca5dd705c7" not found Nov 03 02:08:03.101312 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:08:03.101252 5463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-provider-credentials podName:0ec59c0c-5e04-438c-84a3-1b4f28d8f609 nodeName:}" failed. No retries permitted until 2026-11-03 02:08:11.101233209 +0000 UTC m=+972.785342355 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-provider-credentials") pod "payload-processing-external-model-5db9ffd6f8-2hgf8" (UID: "0ec59c0c-5e04-438c-84a3-1b4f28d8f609") : secret "agc-cred-1dca5dd705c7" not found Nov 03 02:08:03.945604 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:03.945574 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 02:08:03.945993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:03.945632 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" podUID="32ef5394-19c0-4b33-84f6-2087dede03a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 02:08:04.091558 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:04.091522 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 02:08:04.091824 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:04.091795 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" podUID="1e5aa4d8-7e07-4efe-96ef-93b35748423e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 02:08:04.945474 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:04.945430 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 02:08:04.945684 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:04.945503 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" podUID="32ef5394-19c0-4b33-84f6-2087dede03a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 02:08:05.101216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:05.091575 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 02:08:05.101216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:05.091636 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" podUID="1e5aa4d8-7e07-4efe-96ef-93b35748423e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 02:08:05.883220 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:05.882276 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" podUID="0ec59c0c-5e04-438c-84a3-1b4f28d8f609" containerName="payload-processing" containerID="cri-o://fac100833716d141d3d4dbe26b1bee1f04297680eb1710235b554cf3c4314f72" gracePeriod=25 Nov 03 02:08:05.960751 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:08:05.960721 5463 logging.go:55] [core] [Channel #1031 SubChannel #1032]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.78:50052", ServerName: "10.133.0.78:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.78:50052: connect: connection refused" Nov 03 02:08:05.974218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:05.970776 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 02:08:05.974218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:05.970834 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" podUID="32ef5394-19c0-4b33-84f6-2087dede03a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 02:08:06.092431 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:06.091410 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 02:08:06.092431 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:06.091465 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" podUID="1e5aa4d8-7e07-4efe-96ef-93b35748423e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 02:08:06.949217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:06.935463 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" Nov 03 02:08:07.000216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:06.974914 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-kube-api-access-bx89p\" (UniqueName: \"kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-kube-api-access-bx89p\") pod \"0ec59c0c-5e04-438c-84a3-1b4f28d8f609\" (UID: \"0ec59c0c-5e04-438c-84a3-1b4f28d8f609\") " Nov 03 02:08:07.026385 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.026353 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-plugins-config-volume\") pod \"0ec59c0c-5e04-438c-84a3-1b4f28d8f609\" (UID: \"0ec59c0c-5e04-438c-84a3-1b4f28d8f609\") " Nov 03 02:08:07.026703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.026680 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-provider-credentials\") pod \"0ec59c0c-5e04-438c-84a3-1b4f28d8f609\" (UID: \"0ec59c0c-5e04-438c-84a3-1b4f28d8f609\") " Nov 03 02:08:07.026933 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.026914 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-routing-overlay-volume\") pod \"0ec59c0c-5e04-438c-84a3-1b4f28d8f609\" (UID: \"0ec59c0c-5e04-438c-84a3-1b4f28d8f609\") " Nov 03 02:08:07.027931 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:06.975499 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" podUID="0ec59c0c-5e04-438c-84a3-1b4f28d8f609" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.78:50052\" within 1s: context deadline exceeded" Nov 03 02:08:07.028778 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:08:06.975579 5463 logging.go:55] [core] [Channel #1031 SubChannel #1032]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.78:50052", ServerName: "10.133.0.78:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.78:50052: operation was canceled" Nov 03 02:08:07.028778 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:06.975621 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 02:08:07.028778 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.028716 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" podUID="32ef5394-19c0-4b33-84f6-2087dede03a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 02:08:07.028778 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.000914 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-kube-api-access-bx89p" pod "0ec59c0c-5e04-438c-84a3-1b4f28d8f609" (UID: "0ec59c0c-5e04-438c-84a3-1b4f28d8f609"). InnerVolumeSpecName "kube-api-access-bx89p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:08:07.028778 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.027905 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-routing-overlay-volume" pod "0ec59c0c-5e04-438c-84a3-1b4f28d8f609" (UID: "0ec59c0c-5e04-438c-84a3-1b4f28d8f609"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:08:07.029612 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.029588 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-plugins-config-volume" pod "0ec59c0c-5e04-438c-84a3-1b4f28d8f609" (UID: "0ec59c0c-5e04-438c-84a3-1b4f28d8f609"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:08:07.071453 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.064082 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-provider-credentials" pod "0ec59c0c-5e04-438c-84a3-1b4f28d8f609" (UID: "0ec59c0c-5e04-438c-84a3-1b4f28d8f609"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:08:07.071453 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.070554 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" podStartSLOduration=6.070534371 podStartE2EDuration="6.070534371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:08:03.061929513 +0000 UTC m=+964.746038669" watchObservedRunningTime="2026-11-03 02:08:07.070534371 +0000 UTC m=+968.754643529" Nov 03 02:08:07.104222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.097974 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 02:08:07.104222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.098038 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" podUID="1e5aa4d8-7e07-4efe-96ef-93b35748423e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 02:08:07.105258 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.083344 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:08:07.129313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.129276 5463 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-routing-overlay-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:08:07.129544 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.129522 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx89p\" (UniqueName: \"kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-kube-api-access-bx89p\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:08:07.129645 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.129632 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:08:07.129747 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.129734 5463 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ec59c0c-5e04-438c-84a3-1b4f28d8f609-provider-credentials\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:08:07.187606 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.187570 5463 generic.go:356] "Generic (PLEG): container finished" podID="0ec59c0c-5e04-438c-84a3-1b4f28d8f609" containerID="fac100833716d141d3d4dbe26b1bee1f04297680eb1710235b554cf3c4314f72" exitCode=0 Nov 03 02:08:07.188033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.187883 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" Nov 03 02:08:07.189497 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.187905 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" event={"ID":"0ec59c0c-5e04-438c-84a3-1b4f28d8f609","Type":"ContainerDied","Data":"fac100833716d141d3d4dbe26b1bee1f04297680eb1710235b554cf3c4314f72"} Nov 03 02:08:07.190358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.189655 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8" event={"ID":"0ec59c0c-5e04-438c-84a3-1b4f28d8f609","Type":"ContainerDied","Data":"3bd12455c8245bcf195c12e814ce068a5cacb9ff4418b6b84dccc6cc5834b99e"} Nov 03 02:08:07.190358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.189698 5463 scope.go:122] "RemoveContainer" containerID="fac100833716d141d3d4dbe26b1bee1f04297680eb1710235b554cf3c4314f72" Nov 03 02:08:07.210681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.210656 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:07.301609 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.300464 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:08:07.366835 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.366750 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" podUID="0052a06a-6e50-4aa4-9455-b419b9f36f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:08:07.387380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.387355 5463 scope.go:122] "RemoveContainer" containerID="fac100833716d141d3d4dbe26b1bee1f04297680eb1710235b554cf3c4314f72" Nov 03 02:08:07.387994 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:08:07.387970 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac100833716d141d3d4dbe26b1bee1f04297680eb1710235b554cf3c4314f72\": container with ID starting with fac100833716d141d3d4dbe26b1bee1f04297680eb1710235b554cf3c4314f72 not found: ID does not exist" containerID="fac100833716d141d3d4dbe26b1bee1f04297680eb1710235b554cf3c4314f72" Nov 03 02:08:07.388146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.388110 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac100833716d141d3d4dbe26b1bee1f04297680eb1710235b554cf3c4314f72"} err="failed to get container status \"fac100833716d141d3d4dbe26b1bee1f04297680eb1710235b554cf3c4314f72\": rpc error: code = NotFound desc = could not find container \"fac100833716d141d3d4dbe26b1bee1f04297680eb1710235b554cf3c4314f72\": container with ID starting with fac100833716d141d3d4dbe26b1bee1f04297680eb1710235b554cf3c4314f72 not found: ID does not exist" Nov 03 02:08:07.644519 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.644485 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8"] Nov 03 02:08:07.667299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.644746 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 02:08:07.780881 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.780827 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:08:07.802746 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.802708 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db9ffd6f8-2hgf8"] Nov 03 02:08:07.960278 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.956753 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 02:08:07.960278 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:07.956814 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" podUID="32ef5394-19c0-4b33-84f6-2087dede03a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 02:08:08.102232 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:08.101955 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 02:08:08.102232 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:08.102026 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" podUID="1e5aa4d8-7e07-4efe-96ef-93b35748423e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 02:08:08.340453 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:08.340362 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:08:08.378319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:08.378276 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" podUID="0052a06a-6e50-4aa4-9455-b419b9f36f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 02:08:08.379154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:08.379129 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:08:08.475873 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:08.475833 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ec59c0c-5e04-438c-84a3-1b4f28d8f609" path="/var/lib/kubelet/pods/0ec59c0c-5e04-438c-84a3-1b4f28d8f609/volumes" Nov 03 02:08:08.688318 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:08.688274 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 02:08:08.689285 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:08.689260 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:08:08.844214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:08.832728 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:08.920546 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:08.920237 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:08:09.037976 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:09.037861 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 02:08:09.039634 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:09.039589 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" podUID="32ef5394-19c0-4b33-84f6-2087dede03a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 02:08:09.096177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:09.096146 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 02:08:09.096930 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:09.096906 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" podUID="1e5aa4d8-7e07-4efe-96ef-93b35748423e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 02:08:09.097071 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:09.096849 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 02:08:09.314266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:09.298289 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:08:09.402214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:09.380214 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" podUID="0052a06a-6e50-4aa4-9455-b419b9f36f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 02:08:09.703149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:09.703107 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:08:09.756385 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:09.756277 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:08:09.846556 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:09.846420 5463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:08:09.945477 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:09.945444 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 02:08:09.945778 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:09.945735 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" podUID="32ef5394-19c0-4b33-84f6-2087dede03a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 02:08:10.093088 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:10.092991 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 02:08:10.093369 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:10.093326 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" podUID="1e5aa4d8-7e07-4efe-96ef-93b35748423e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 02:08:10.098377 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:10.098337 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:08:10.098674 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:10.098646 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:08:10.326494 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:10.326432 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:08:10.365306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:10.341081 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:08:10.365306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:10.342860 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:08:10.384018 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:10.383268 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" podUID="0052a06a-6e50-4aa4-9455-b419b9f36f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:08:10.449565 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:10.449528 5463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:08:10.744923 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:10.744881 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 02:08:10.782496 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:10.777933 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 03 02:08:10.787209 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:10.783827 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:08:10.787209 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:10.783892 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:08:10.787209 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:10.785465 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:08:10.891147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:10.890412 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:08:11.254222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:11.251796 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:08:11.256661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:11.256624 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:08:11.294067 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:11.294020 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 02:08:11.294386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:11.294353 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" podUID="1e5aa4d8-7e07-4efe-96ef-93b35748423e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 02:08:11.382273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:11.378282 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:11.419216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:11.398511 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:11.419216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:11.408397 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:08:11.426239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:11.425741 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:08:11.426728 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:11.426705 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:08:11.614690 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:11.614620 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:11.804974 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:11.804933 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:12.131603 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:12.131573 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:12.215994 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:12.215964 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:12.216472 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:12.216436 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:12.217567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:12.217544 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:08:12.412858 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:12.412830 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:13.008116 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:08:13.008082 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:08:14.331596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.331557 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf"] Nov 03 02:08:14.332323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.332303 5463 cpu_manager.go:414] "containerMap: removing container" podUID="0ec59c0c-5e04-438c-84a3-1b4f28d8f609" containerName="payload-processing" Nov 03 02:08:14.332323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.332324 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec59c0c-5e04-438c-84a3-1b4f28d8f609" containerName="payload-processing" Nov 03 02:08:14.332520 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.332507 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec59c0c-5e04-438c-84a3-1b4f28d8f609" containerName="payload-processing" Nov 03 02:08:14.335743 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.335723 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.388657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.388629 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-521f1f-openshift-default-dockercfg-c9pnp\"" Nov 03 02:08:14.445311 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.445286 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf"] Nov 03 02:08:14.493160 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.493122 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.493335 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.493238 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.493335 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.493265 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.493335 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.493283 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.493335 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.493322 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.493557 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.493407 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.493557 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.493443 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.493557 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.493518 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.493557 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.493541 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zzjg\" (UniqueName: \"kubernetes.io/projected/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-kube-api-access-7zzjg\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.595245 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.595158 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.595245 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.595217 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.595453 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.595276 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.595453 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.595410 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zzjg\" (UniqueName: \"kubernetes.io/projected/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-kube-api-access-7zzjg\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.595564 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.595521 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.595564 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.595543 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.595675 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.595586 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.595675 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.595633 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.595675 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.595661 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.595831 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.595692 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.595831 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.595743 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.595974 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.595954 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.596145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.596090 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.596400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.596379 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.597685 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.597664 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.597814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.597796 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.652543 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.652512 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.652644 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.652576 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zzjg\" (UniqueName: \"kubernetes.io/projected/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-kube-api-access-7zzjg\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:14.948798 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:14.948759 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:15.368741 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:08:15.368709 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfeb7b50_ad6f_4fa2_b672_cb47242deb1e.slice/crio-87a2e7fd42de7d07a584ecb549b719aab25bc2e4bb5b84d6b57068c0475e7e82 WatchSource:0}: Error finding container 87a2e7fd42de7d07a584ecb549b719aab25bc2e4bb5b84d6b57068c0475e7e82: Status 404 returned error can't find the container with id 87a2e7fd42de7d07a584ecb549b719aab25bc2e4bb5b84d6b57068c0475e7e82 Nov 03 02:08:15.371480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:15.371441 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:08:15.371597 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:15.371530 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:08:15.371597 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:15.371573 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:08:15.372616 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:15.372597 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf"] Nov 03 02:08:15.607234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:15.607167 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" event={"ID":"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e","Type":"ContainerStarted","Data":"01c1dd81d2ac1849ef17d16215b7b100b407a0698d4d277555af1db42e47d994"} Nov 03 02:08:15.607417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:15.607265 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" event={"ID":"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e","Type":"ContainerStarted","Data":"87a2e7fd42de7d07a584ecb549b719aab25bc2e4bb5b84d6b57068c0475e7e82"} Nov 03 02:08:15.725585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:15.725533 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" podStartSLOduration=1.725519142 podStartE2EDuration="1.725519142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:08:15.722598659 +0000 UTC m=+977.406707830" watchObservedRunningTime="2026-11-03 02:08:15.725519142 +0000 UTC m=+977.409628296" Nov 03 02:08:15.952214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:15.950269 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:16.827220 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:16.827172 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:16.964215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:16.952296 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 02:08:16.964215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:16.952351 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" podUID="dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 02:08:17.955218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:17.951491 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 02:08:17.955218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:17.951557 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" podUID="dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 02:08:18.982918 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:18.982884 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 02:08:18.983466 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:18.983434 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" podUID="dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 02:08:19.957319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:19.957281 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 02:08:19.957566 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:19.957540 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" podUID="dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 02:08:20.967215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:20.958508 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 02:08:20.967215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:20.958567 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" podUID="dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 02:08:21.525908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:21.525869 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:21.949562 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:21.949523 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 02:08:21.949764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:21.949641 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" podUID="dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 02:08:22.955179 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:22.955143 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:22.955712 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:22.955691 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:22.956317 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:22.956292 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:08:26.379405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:26.379364 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:31.031454 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:31.031415 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:35.733175 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:35.733131 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:40.513633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:40.513556 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:43.007302 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:08:43.007260 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:08:45.324435 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:45.324307 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:50.131821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:50.131782 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:52.945330 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:52.945263 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:54.857825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:54.857796 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:58.695906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:58.695875 5463 scope.go:122] "RemoveContainer" containerID="50453e98c6dcffb1d644880d61d215b9360a9a348db198c212e3f95123964ad6" Nov 03 02:08:59.606810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:08:59.606775 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:04.351233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:04.351145 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:09.045443 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:09.045405 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:09.213532 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:09.213502 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:10.263886 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:10.263853 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:13.007308 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:09:13.007265 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:09:13.725032 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:13.724998 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:18.432711 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:18.432671 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:23.224954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:23.224910 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:27.938760 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:27.938725 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:31.279552 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.279522 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7"] Nov 03 02:09:31.283978 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.283958 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.378234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.378156 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e969fe-f189-40f2-b929-e462f6d4b86b-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.378234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.378234 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.378488 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.378327 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.378488 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.378387 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.378488 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.378413 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb96n\" (UniqueName: \"kubernetes.io/projected/13e969fe-f189-40f2-b929-e462f6d4b86b-kube-api-access-fb96n\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.378488 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.378464 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.378696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.378619 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.378696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.378644 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.378805 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.378735 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.479729 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.479695 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.479729 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.479730 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb96n\" (UniqueName: \"kubernetes.io/projected/13e969fe-f189-40f2-b929-e462f6d4b86b-kube-api-access-fb96n\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.479954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.479761 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.479954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.479804 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.479954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.479820 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.479954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.479885 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.480167 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.479963 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e969fe-f189-40f2-b929-e462f6d4b86b-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.480167 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.479985 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.480167 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.480040 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.480427 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.480303 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.480427 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.480355 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.480427 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.480376 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.480553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.480511 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.480882 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.480857 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e969fe-f189-40f2-b929-e462f6d4b86b-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.482352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.482327 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.482461 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.482408 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.519560 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.519526 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7"] Nov 03 02:09:31.544542 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.544477 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.544659 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.544553 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb96n\" (UniqueName: \"kubernetes.io/projected/13e969fe-f189-40f2-b929-e462f6d4b86b-kube-api-access-fb96n\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:31.601804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:31.601777 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:32.015983 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:09:32.015951 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e969fe_f189_40f2_b929_e462f6d4b86b.slice/crio-59af71994020b2bed3ec77c86d1dbe46dba12fc875e0bf46ad58313a73c51184 WatchSource:0}: Error finding container 59af71994020b2bed3ec77c86d1dbe46dba12fc875e0bf46ad58313a73c51184: Status 404 returned error can't find the container with id 59af71994020b2bed3ec77c86d1dbe46dba12fc875e0bf46ad58313a73c51184 Nov 03 02:09:32.018387 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:32.018352 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:09:32.018471 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:32.018420 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:09:32.018471 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:32.018450 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:09:32.019355 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:32.019337 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7"] Nov 03 02:09:32.167642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:32.167605 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" event={"ID":"13e969fe-f189-40f2-b929-e462f6d4b86b","Type":"ContainerStarted","Data":"141f045aa4d400f57e8e3bf8e8ce8f30fe30ae85ee65e7fce4e42303090a2003"} Nov 03 02:09:32.167814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:32.167666 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" event={"ID":"13e969fe-f189-40f2-b929-e462f6d4b86b","Type":"ContainerStarted","Data":"59af71994020b2bed3ec77c86d1dbe46dba12fc875e0bf46ad58313a73c51184"} Nov 03 02:09:32.286638 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:32.286541 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" podStartSLOduration=1.286525022 podStartE2EDuration="1.286525022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:09:32.283440386 +0000 UTC m=+1053.967549542" watchObservedRunningTime="2026-11-03 02:09:32.286525022 +0000 UTC m=+1053.970634180" Nov 03 02:09:32.602741 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:32.602660 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:32.634877 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:32.634848 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:32.896276 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:32.896245 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:33.604178 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:33.604144 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 02:09:33.604594 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:33.604217 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" podUID="13e969fe-f189-40f2-b929-e462f6d4b86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 02:09:34.602236 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:34.602183 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 02:09:34.602407 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:34.602255 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" podUID="13e969fe-f189-40f2-b929-e462f6d4b86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 02:09:35.082842 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:35.082803 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2"] Nov 03 02:09:35.086921 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:35.086890 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:35.602812 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:35.602666 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 02:09:35.602812 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:35.602751 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" podUID="13e969fe-f189-40f2-b929-e462f6d4b86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 02:09:36.603009 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:36.602971 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 02:09:36.603443 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:36.603029 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" podUID="13e969fe-f189-40f2-b929-e462f6d4b86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 02:09:37.338103 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:37.338072 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:37.602575 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:37.602482 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 02:09:37.602575 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:37.602548 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" podUID="13e969fe-f189-40f2-b929-e462f6d4b86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 02:09:38.602580 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:38.602540 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 02:09:38.603096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:38.602597 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" podUID="13e969fe-f189-40f2-b929-e462f6d4b86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 02:09:39.608797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:39.608761 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:39.609914 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:39.609890 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:39.610472 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:39.610453 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:09:39.663843 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:39.663820 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:39.851835 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:39.851799 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:42.035568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:42.035539 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:43.007549 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:09:43.007519 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:09:46.727554 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:46.727520 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:47.525227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.523395 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6"] Nov 03 02:09:47.530430 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.530403 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.600267 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.600227 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jx56\" (UniqueName: \"kubernetes.io/projected/8d2c4397-7a8c-41a5-9140-2079f97bd460-kube-api-access-7jx56\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.600442 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.600395 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.600442 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.600431 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.600569 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.600514 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.600569 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.600554 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.600666 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.600599 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.600666 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.600624 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.600666 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.600662 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d2c4397-7a8c-41a5-9140-2079f97bd460-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.600818 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.600678 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.702352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.702316 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jx56\" (UniqueName: \"kubernetes.io/projected/8d2c4397-7a8c-41a5-9140-2079f97bd460-kube-api-access-7jx56\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.702518 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.702400 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.702518 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.702441 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.702607 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.702533 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.702607 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.702580 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.702710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.702642 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.702710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.702681 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.702822 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.702731 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d2c4397-7a8c-41a5-9140-2079f97bd460-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.702822 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.702759 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.702822 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.702786 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.702975 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.702872 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.703039 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.703012 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.703113 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.703090 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.703475 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.703447 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d2c4397-7a8c-41a5-9140-2079f97bd460-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.705125 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.705099 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.705286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.705243 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.723141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.723112 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6"] Nov 03 02:09:47.759632 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.759602 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.760053 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.759940 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jx56\" (UniqueName: \"kubernetes.io/projected/8d2c4397-7a8c-41a5-9140-2079f97bd460-kube-api-access-7jx56\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:47.846138 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:47.846066 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:48.267457 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:09:48.267424 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2c4397_7a8c_41a5_9140_2079f97bd460.slice/crio-11506ee390343020b2cd9a2ce94c0a6efbdccfd6525e47fa4c710a8d43f5ea72 WatchSource:0}: Error finding container 11506ee390343020b2cd9a2ce94c0a6efbdccfd6525e47fa4c710a8d43f5ea72: Status 404 returned error can't find the container with id 11506ee390343020b2cd9a2ce94c0a6efbdccfd6525e47fa4c710a8d43f5ea72 Nov 03 02:09:48.270255 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:48.270234 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6"] Nov 03 02:09:48.270614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:48.270584 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:09:48.270663 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:48.270648 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:09:48.270698 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:48.270679 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:09:48.319573 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:48.319541 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" event={"ID":"8d2c4397-7a8c-41a5-9140-2079f97bd460","Type":"ContainerStarted","Data":"11506ee390343020b2cd9a2ce94c0a6efbdccfd6525e47fa4c710a8d43f5ea72"} Nov 03 02:09:49.327986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:49.327944 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" event={"ID":"8d2c4397-7a8c-41a5-9140-2079f97bd460","Type":"ContainerStarted","Data":"cbbbb46a6aa43d10924278e813715e80a88daa2a5e98c1df1f9a635c078c6c40"} Nov 03 02:09:49.545712 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:49.545657 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" podStartSLOduration=2.545645278 podStartE2EDuration="2.545645278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:09:49.495448265 +0000 UTC m=+1071.179557419" watchObservedRunningTime="2026-11-03 02:09:49.545645278 +0000 UTC m=+1071.229754434" Nov 03 02:09:49.847165 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:49.847135 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:49.870212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:49.852327 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 02:09:49.870212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:49.852400 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" podUID="8d2c4397-7a8c-41a5-9140-2079f97bd460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 02:09:50.103945 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.103859 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" podUID="0052a06a-6e50-4aa4-9455-b419b9f36f34" containerName="main" containerID="cri-o://9ef5bf755ec5aeeb77dc449d19b6012a84b349ae7fdf5b0f29ec2529437d8337" gracePeriod=45 Nov 03 02:09:50.403395 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.403360 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" podUID="0052a06a-6e50-4aa4-9455-b419b9f36f34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": dial tcp 10.133.0.82:8000: connect: connection refused" Nov 03 02:09:50.408789 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.407712 5463 generic.go:356] "Generic (PLEG): container finished" podID="0052a06a-6e50-4aa4-9455-b419b9f36f34" containerID="9ef5bf755ec5aeeb77dc449d19b6012a84b349ae7fdf5b0f29ec2529437d8337" exitCode=0 Nov 03 02:09:50.411844 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.411786 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" event={"ID":"0052a06a-6e50-4aa4-9455-b419b9f36f34","Type":"ContainerDied","Data":"9ef5bf755ec5aeeb77dc449d19b6012a84b349ae7fdf5b0f29ec2529437d8337"} Nov 03 02:09:50.726847 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.726822 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:09:50.834785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.834519 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-home\" (UniqueName: \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-home\") pod \"0052a06a-6e50-4aa4-9455-b419b9f36f34\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " Nov 03 02:09:50.834785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.834623 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-dshm\") pod \"0052a06a-6e50-4aa4-9455-b419b9f36f34\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " Nov 03 02:09:50.834785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.834654 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0052a06a-6e50-4aa4-9455-b419b9f36f34-kube-api-access-wjnpl\" (UniqueName: \"kubernetes.io/projected/0052a06a-6e50-4aa4-9455-b419b9f36f34-kube-api-access-wjnpl\") pod \"0052a06a-6e50-4aa4-9455-b419b9f36f34\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " Nov 03 02:09:50.834785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.834720 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-model-cache\") pod \"0052a06a-6e50-4aa4-9455-b419b9f36f34\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " Nov 03 02:09:50.834785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.834742 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-home" pod "0052a06a-6e50-4aa4-9455-b419b9f36f34" (UID: "0052a06a-6e50-4aa4-9455-b419b9f36f34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:09:50.834785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.834764 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0052a06a-6e50-4aa4-9455-b419b9f36f34-tls-certs\" (UniqueName: \"kubernetes.io/secret/0052a06a-6e50-4aa4-9455-b419b9f36f34-tls-certs\") pod \"0052a06a-6e50-4aa4-9455-b419b9f36f34\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " Nov 03 02:09:50.835285 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.834809 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-tmp-dir\") pod \"0052a06a-6e50-4aa4-9455-b419b9f36f34\" (UID: \"0052a06a-6e50-4aa4-9455-b419b9f36f34\") " Nov 03 02:09:50.835478 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.835455 5463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-home\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:50.835567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.835488 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-model-cache" pod "0052a06a-6e50-4aa4-9455-b419b9f36f34" (UID: "0052a06a-6e50-4aa4-9455-b419b9f36f34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:09:50.835679 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.835658 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-tmp-dir" pod "0052a06a-6e50-4aa4-9455-b419b9f36f34" (UID: "0052a06a-6e50-4aa4-9455-b419b9f36f34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:09:50.837136 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.837068 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-dshm" pod "0052a06a-6e50-4aa4-9455-b419b9f36f34" (UID: "0052a06a-6e50-4aa4-9455-b419b9f36f34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:09:50.837136 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.837102 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0052a06a-6e50-4aa4-9455-b419b9f36f34-kube-api-access-wjnpl" pod "0052a06a-6e50-4aa4-9455-b419b9f36f34" (UID: "0052a06a-6e50-4aa4-9455-b419b9f36f34"). InnerVolumeSpecName "kube-api-access-wjnpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:09:50.837953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.837926 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0052a06a-6e50-4aa4-9455-b419b9f36f34-tls-certs" pod "0052a06a-6e50-4aa4-9455-b419b9f36f34" (UID: "0052a06a-6e50-4aa4-9455-b419b9f36f34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:09:50.882259 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.847369 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 02:09:50.882259 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.847419 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" podUID="8d2c4397-7a8c-41a5-9140-2079f97bd460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 02:09:50.941419 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.937000 5463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-dshm\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:50.941419 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.937025 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjnpl\" (UniqueName: \"kubernetes.io/projected/0052a06a-6e50-4aa4-9455-b419b9f36f34-kube-api-access-wjnpl\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:50.941419 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.937035 5463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-model-cache\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:50.941419 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.937044 5463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0052a06a-6e50-4aa4-9455-b419b9f36f34-tls-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:50.941419 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:50.937053 5463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0052a06a-6e50-4aa4-9455-b419b9f36f34-tmp-dir\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:51.435355 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:51.435305 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" event={"ID":"0052a06a-6e50-4aa4-9455-b419b9f36f34","Type":"ContainerDied","Data":"a372de8aa6e9da7f83949f56e0c2b8a2504b857973595c6d1dd54a6dac555ae6"} Nov 03 02:09:51.435834 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:51.435367 5463 scope.go:122] "RemoveContainer" containerID="9ef5bf755ec5aeeb77dc449d19b6012a84b349ae7fdf5b0f29ec2529437d8337" Nov 03 02:09:51.435834 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:51.435667 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2" Nov 03 02:09:51.608266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:51.608236 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2"] Nov 03 02:09:51.613413 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:51.613385 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/e2e-conflict-a-7e89b72d-kserve-784d5d6bcc-6r4q2"] Nov 03 02:09:51.806425 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:51.806341 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:51.847136 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:51.847093 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 02:09:51.847381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:51.847164 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" podUID="8d2c4397-7a8c-41a5-9140-2079f97bd460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 02:09:52.443906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:52.443874 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0052a06a-6e50-4aa4-9455-b419b9f36f34" path="/var/lib/kubelet/pods/0052a06a-6e50-4aa4-9455-b419b9f36f34/volumes" Nov 03 02:09:52.847539 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:52.847454 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 02:09:52.847795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:52.847763 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" podUID="8d2c4397-7a8c-41a5-9140-2079f97bd460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 02:09:53.744863 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.744823 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk"] Nov 03 02:09:53.745629 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.745606 5463 cpu_manager.go:414] "containerMap: removing container" podUID="0052a06a-6e50-4aa4-9455-b419b9f36f34" containerName="main" Nov 03 02:09:53.745731 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.745631 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0052a06a-6e50-4aa4-9455-b419b9f36f34" containerName="main" Nov 03 02:09:53.745791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.745784 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0052a06a-6e50-4aa4-9455-b419b9f36f34" containerName="main" Nov 03 02:09:53.768216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.768140 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk" Nov 03 02:09:53.802540 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.802510 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk"] Nov 03 02:09:53.821763 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.821733 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d23af3-99d2-4694-ae47-aff7d32796ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26fcf768-sm7pk\" (UID: \"b7d23af3-99d2-4694-ae47-aff7d32796ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk" Nov 03 02:09:53.821926 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.821813 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m74mt\" (UniqueName: \"kubernetes.io/projected/b7d23af3-99d2-4694-ae47-aff7d32796ff-kube-api-access-m74mt\") pod \"maas-api-revoke-sub-26fcf768-sm7pk\" (UID: \"b7d23af3-99d2-4694-ae47-aff7d32796ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk" Nov 03 02:09:53.821926 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.821837 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d23af3-99d2-4694-ae47-aff7d32796ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26fcf768-sm7pk\" (UID: \"b7d23af3-99d2-4694-ae47-aff7d32796ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk" Nov 03 02:09:53.847112 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.847082 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 02:09:53.847304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.847137 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" podUID="8d2c4397-7a8c-41a5-9140-2079f97bd460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 02:09:53.872241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.871578 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:09:53.872241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.871785 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:09:53.872241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.871876 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:09:53.872241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.871963 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:09:53.922728 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.922698 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d23af3-99d2-4694-ae47-aff7d32796ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26fcf768-sm7pk\" (UID: \"b7d23af3-99d2-4694-ae47-aff7d32796ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk" Nov 03 02:09:53.922895 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.922777 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m74mt\" (UniqueName: \"kubernetes.io/projected/b7d23af3-99d2-4694-ae47-aff7d32796ff-kube-api-access-m74mt\") pod \"maas-api-revoke-sub-26fcf768-sm7pk\" (UID: \"b7d23af3-99d2-4694-ae47-aff7d32796ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk" Nov 03 02:09:53.922895 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.922805 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d23af3-99d2-4694-ae47-aff7d32796ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26fcf768-sm7pk\" (UID: \"b7d23af3-99d2-4694-ae47-aff7d32796ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk" Nov 03 02:09:53.923465 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.923439 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d23af3-99d2-4694-ae47-aff7d32796ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26fcf768-sm7pk\" (UID: \"b7d23af3-99d2-4694-ae47-aff7d32796ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk" Nov 03 02:09:53.925121 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.925097 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d23af3-99d2-4694-ae47-aff7d32796ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26fcf768-sm7pk\" (UID: \"b7d23af3-99d2-4694-ae47-aff7d32796ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk" Nov 03 02:09:53.980618 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:53.980590 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m74mt\" (UniqueName: \"kubernetes.io/projected/b7d23af3-99d2-4694-ae47-aff7d32796ff-kube-api-access-m74mt\") pod \"maas-api-revoke-sub-26fcf768-sm7pk\" (UID: \"b7d23af3-99d2-4694-ae47-aff7d32796ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk" Nov 03 02:09:54.085896 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:54.085818 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk" Nov 03 02:09:54.477747 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:54.477355 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk"] Nov 03 02:09:54.480325 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:09:54.480296 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d23af3_99d2_4694_ae47_aff7d32796ff.slice/crio-4f0d52f9e1629b79cdf1b90ee9128731a3ee4c8c5a53976d555293d4320ec8bb WatchSource:0}: Error finding container 4f0d52f9e1629b79cdf1b90ee9128731a3ee4c8c5a53976d555293d4320ec8bb: Status 404 returned error can't find the container with id 4f0d52f9e1629b79cdf1b90ee9128731a3ee4c8c5a53976d555293d4320ec8bb Nov 03 02:09:54.847053 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:54.846977 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 02:09:54.847053 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:54.847030 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" podUID="8d2c4397-7a8c-41a5-9140-2079f97bd460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 02:09:55.496979 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:55.496942 5463 generic.go:356] "Generic (PLEG): container finished" podID="b7d23af3-99d2-4694-ae47-aff7d32796ff" containerID="24c9c5f9633337e7c66fdf5652b82bd7fe96d61126dbdf8d717c4e93b996a2c3" exitCode=0 Nov 03 02:09:55.497174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:55.497034 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk" event={"ID":"b7d23af3-99d2-4694-ae47-aff7d32796ff","Type":"ContainerDied","Data":"24c9c5f9633337e7c66fdf5652b82bd7fe96d61126dbdf8d717c4e93b996a2c3"} Nov 03 02:09:55.497174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:55.497129 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk" event={"ID":"b7d23af3-99d2-4694-ae47-aff7d32796ff","Type":"ContainerStarted","Data":"4f0d52f9e1629b79cdf1b90ee9128731a3ee4c8c5a53976d555293d4320ec8bb"} Nov 03 02:09:55.945949 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:55.945914 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:55.946684 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:55.946644 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:55.947895 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:55.947876 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:09:56.658393 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:56.658370 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk" Nov 03 02:09:56.758720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:56.758644 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d23af3-99d2-4694-ae47-aff7d32796ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d23af3-99d2-4694-ae47-aff7d32796ff-maas-api-openshift-ca\") pod \"b7d23af3-99d2-4694-ae47-aff7d32796ff\" (UID: \"b7d23af3-99d2-4694-ae47-aff7d32796ff\") " Nov 03 02:09:56.758873 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:56.758814 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d23af3-99d2-4694-ae47-aff7d32796ff-kube-api-access-m74mt\" (UniqueName: \"kubernetes.io/projected/b7d23af3-99d2-4694-ae47-aff7d32796ff-kube-api-access-m74mt\") pod \"b7d23af3-99d2-4694-ae47-aff7d32796ff\" (UID: \"b7d23af3-99d2-4694-ae47-aff7d32796ff\") " Nov 03 02:09:56.758873 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:56.758848 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7d23af3-99d2-4694-ae47-aff7d32796ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d23af3-99d2-4694-ae47-aff7d32796ff-maas-api-xks-ca\") pod \"b7d23af3-99d2-4694-ae47-aff7d32796ff\" (UID: \"b7d23af3-99d2-4694-ae47-aff7d32796ff\") " Nov 03 02:09:56.758985 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:56.758946 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d23af3-99d2-4694-ae47-aff7d32796ff-maas-api-openshift-ca" pod "b7d23af3-99d2-4694-ae47-aff7d32796ff" (UID: "b7d23af3-99d2-4694-ae47-aff7d32796ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:09:56.759249 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:56.759232 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d23af3-99d2-4694-ae47-aff7d32796ff-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:56.760874 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:56.760854 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d23af3-99d2-4694-ae47-aff7d32796ff-maas-api-xks-ca" pod "b7d23af3-99d2-4694-ae47-aff7d32796ff" (UID: "b7d23af3-99d2-4694-ae47-aff7d32796ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:09:56.760945 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:56.760889 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d23af3-99d2-4694-ae47-aff7d32796ff-kube-api-access-m74mt" pod "b7d23af3-99d2-4694-ae47-aff7d32796ff" (UID: "b7d23af3-99d2-4694-ae47-aff7d32796ff"). InnerVolumeSpecName "kube-api-access-m74mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:09:56.824738 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:56.824709 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:56.861025 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:56.860989 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m74mt\" (UniqueName: \"kubernetes.io/projected/b7d23af3-99d2-4694-ae47-aff7d32796ff-kube-api-access-m74mt\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:56.861025 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:56.861022 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d23af3-99d2-4694-ae47-aff7d32796ff-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:57.511994 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.511958 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk" event={"ID":"b7d23af3-99d2-4694-ae47-aff7d32796ff","Type":"ContainerDied","Data":"4f0d52f9e1629b79cdf1b90ee9128731a3ee4c8c5a53976d555293d4320ec8bb"} Nov 03 02:09:57.512428 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.512019 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f0d52f9e1629b79cdf1b90ee9128731a3ee4c8c5a53976d555293d4320ec8bb" Nov 03 02:09:57.512428 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.511975 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk" Nov 03 02:09:57.572242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.572215 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc"] Nov 03 02:09:57.572858 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.572831 5463 cpu_manager.go:414] "containerMap: removing container" podUID="b7d23af3-99d2-4694-ae47-aff7d32796ff" containerName="revoke-keys" Nov 03 02:09:57.572858 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.572855 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d23af3-99d2-4694-ae47-aff7d32796ff" containerName="revoke-keys" Nov 03 02:09:57.573048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.572954 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d23af3-99d2-4694-ae47-aff7d32796ff" containerName="revoke-keys" Nov 03 02:09:57.576487 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.576468 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc" Nov 03 02:09:57.629214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.629174 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc"] Nov 03 02:09:57.631248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.630288 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:09:57.631248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.630314 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:09:57.631248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.630409 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:09:57.631248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.630555 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:09:57.670470 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.670426 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5877bb76-8962-42bd-b4e6-5b1ac38978d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20177b10-r27pc\" (UID: \"5877bb76-8962-42bd-b4e6-5b1ac38978d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc" Nov 03 02:09:57.670666 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.670638 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdlgb\" (UniqueName: \"kubernetes.io/projected/5877bb76-8962-42bd-b4e6-5b1ac38978d2-kube-api-access-vdlgb\") pod \"maas-api-revoke-sub-20177b10-r27pc\" (UID: \"5877bb76-8962-42bd-b4e6-5b1ac38978d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc" Nov 03 02:09:57.670733 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.670697 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5877bb76-8962-42bd-b4e6-5b1ac38978d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20177b10-r27pc\" (UID: \"5877bb76-8962-42bd-b4e6-5b1ac38978d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc" Nov 03 02:09:57.771936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.771840 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5877bb76-8962-42bd-b4e6-5b1ac38978d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20177b10-r27pc\" (UID: \"5877bb76-8962-42bd-b4e6-5b1ac38978d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc" Nov 03 02:09:57.772088 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.771940 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdlgb\" (UniqueName: \"kubernetes.io/projected/5877bb76-8962-42bd-b4e6-5b1ac38978d2-kube-api-access-vdlgb\") pod \"maas-api-revoke-sub-20177b10-r27pc\" (UID: \"5877bb76-8962-42bd-b4e6-5b1ac38978d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc" Nov 03 02:09:57.772088 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.771965 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5877bb76-8962-42bd-b4e6-5b1ac38978d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20177b10-r27pc\" (UID: \"5877bb76-8962-42bd-b4e6-5b1ac38978d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc" Nov 03 02:09:57.772721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.772694 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5877bb76-8962-42bd-b4e6-5b1ac38978d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20177b10-r27pc\" (UID: \"5877bb76-8962-42bd-b4e6-5b1ac38978d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc" Nov 03 02:09:57.774356 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.774334 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5877bb76-8962-42bd-b4e6-5b1ac38978d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20177b10-r27pc\" (UID: \"5877bb76-8962-42bd-b4e6-5b1ac38978d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc" Nov 03 02:09:57.832099 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.832069 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdlgb\" (UniqueName: \"kubernetes.io/projected/5877bb76-8962-42bd-b4e6-5b1ac38978d2-kube-api-access-vdlgb\") pod \"maas-api-revoke-sub-20177b10-r27pc\" (UID: \"5877bb76-8962-42bd-b4e6-5b1ac38978d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc" Nov 03 02:09:57.890503 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:57.890472 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc" Nov 03 02:09:58.259621 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:09:58.259588 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5877bb76_8962_42bd_b4e6_5b1ac38978d2.slice/crio-c379ad657a6f6d4aaeb22712d70a3189d622c56fe2a69d8ad7a9dece82c09ff3 WatchSource:0}: Error finding container c379ad657a6f6d4aaeb22712d70a3189d622c56fe2a69d8ad7a9dece82c09ff3: Status 404 returned error can't find the container with id c379ad657a6f6d4aaeb22712d70a3189d622c56fe2a69d8ad7a9dece82c09ff3 Nov 03 02:09:58.261946 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:58.261919 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc"] Nov 03 02:09:58.519489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:58.519425 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc" event={"ID":"5877bb76-8962-42bd-b4e6-5b1ac38978d2","Type":"ContainerStarted","Data":"c379ad657a6f6d4aaeb22712d70a3189d622c56fe2a69d8ad7a9dece82c09ff3"} Nov 03 02:09:59.528548 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:59.528514 5463 generic.go:356] "Generic (PLEG): container finished" podID="5877bb76-8962-42bd-b4e6-5b1ac38978d2" containerID="9f499029bbce2c2ec0de400dd61f1b632396644b908a6b9c3a36b808591532c7" exitCode=0 Nov 03 02:09:59.528951 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:09:59.528590 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc" event={"ID":"5877bb76-8962-42bd-b4e6-5b1ac38978d2","Type":"ContainerDied","Data":"9f499029bbce2c2ec0de400dd61f1b632396644b908a6b9c3a36b808591532c7"} Nov 03 02:10:00.708132 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:00.708107 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc" Nov 03 02:10:00.814077 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:00.814006 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5877bb76-8962-42bd-b4e6-5b1ac38978d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5877bb76-8962-42bd-b4e6-5b1ac38978d2-maas-api-xks-ca\") pod \"5877bb76-8962-42bd-b4e6-5b1ac38978d2\" (UID: \"5877bb76-8962-42bd-b4e6-5b1ac38978d2\") " Nov 03 02:10:00.814077 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:00.814050 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5877bb76-8962-42bd-b4e6-5b1ac38978d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5877bb76-8962-42bd-b4e6-5b1ac38978d2-maas-api-openshift-ca\") pod \"5877bb76-8962-42bd-b4e6-5b1ac38978d2\" (UID: \"5877bb76-8962-42bd-b4e6-5b1ac38978d2\") " Nov 03 02:10:00.814336 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:00.814142 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5877bb76-8962-42bd-b4e6-5b1ac38978d2-kube-api-access-vdlgb\" (UniqueName: \"kubernetes.io/projected/5877bb76-8962-42bd-b4e6-5b1ac38978d2-kube-api-access-vdlgb\") pod \"5877bb76-8962-42bd-b4e6-5b1ac38978d2\" (UID: \"5877bb76-8962-42bd-b4e6-5b1ac38978d2\") " Nov 03 02:10:00.814493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:00.814472 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5877bb76-8962-42bd-b4e6-5b1ac38978d2-maas-api-openshift-ca" pod "5877bb76-8962-42bd-b4e6-5b1ac38978d2" (UID: "5877bb76-8962-42bd-b4e6-5b1ac38978d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:10:00.816142 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:00.816116 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5877bb76-8962-42bd-b4e6-5b1ac38978d2-kube-api-access-vdlgb" pod "5877bb76-8962-42bd-b4e6-5b1ac38978d2" (UID: "5877bb76-8962-42bd-b4e6-5b1ac38978d2"). InnerVolumeSpecName "kube-api-access-vdlgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:00.816142 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:00.816129 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5877bb76-8962-42bd-b4e6-5b1ac38978d2-maas-api-xks-ca" pod "5877bb76-8962-42bd-b4e6-5b1ac38978d2" (UID: "5877bb76-8962-42bd-b4e6-5b1ac38978d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:10:00.915690 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:00.915659 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5877bb76-8962-42bd-b4e6-5b1ac38978d2-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:00.915690 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:00.915686 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5877bb76-8962-42bd-b4e6-5b1ac38978d2-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:00.915867 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:00.915697 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdlgb\" (UniqueName: \"kubernetes.io/projected/5877bb76-8962-42bd-b4e6-5b1ac38978d2-kube-api-access-vdlgb\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:01.536648 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:01.536609 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:01.542550 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:01.542526 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc" Nov 03 02:10:01.542692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:01.542574 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc" event={"ID":"5877bb76-8962-42bd-b4e6-5b1ac38978d2","Type":"ContainerDied","Data":"c379ad657a6f6d4aaeb22712d70a3189d622c56fe2a69d8ad7a9dece82c09ff3"} Nov 03 02:10:01.542692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:01.542621 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c379ad657a6f6d4aaeb22712d70a3189d622c56fe2a69d8ad7a9dece82c09ff3" Nov 03 02:10:05.438216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:05.437812 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 02:10:06.433596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:06.433562 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:08.322529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:08.322492 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d"] Nov 03 02:10:08.323073 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:08.323059 5463 cpu_manager.go:414] "containerMap: removing container" podUID="5877bb76-8962-42bd-b4e6-5b1ac38978d2" containerName="revoke-keys" Nov 03 02:10:08.323115 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:08.323074 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5877bb76-8962-42bd-b4e6-5b1ac38978d2" containerName="revoke-keys" Nov 03 02:10:08.323217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:08.323181 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="5877bb76-8962-42bd-b4e6-5b1ac38978d2" containerName="revoke-keys" Nov 03 02:10:08.326846 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:08.326829 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d" Nov 03 02:10:08.380481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:08.380450 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d"] Nov 03 02:10:08.425904 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:08.425877 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:10:08.429600 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:08.429580 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:10:08.429739 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:08.429638 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:10:08.429739 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:08.429679 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:10:08.456686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:08.456660 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661ba085-9a33-4617-a663-5deba830d1fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dfbda7e-7h26d\" (UID: \"661ba085-9a33-4617-a663-5deba830d1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d" Nov 03 02:10:08.456805 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:08.456708 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661ba085-9a33-4617-a663-5deba830d1fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dfbda7e-7h26d\" (UID: \"661ba085-9a33-4617-a663-5deba830d1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d" Nov 03 02:10:08.456872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:08.456849 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7c4j\" (UniqueName: \"kubernetes.io/projected/661ba085-9a33-4617-a663-5deba830d1fd-kube-api-access-d7c4j\") pod \"maas-api-revoke-sub-7dfbda7e-7h26d\" (UID: \"661ba085-9a33-4617-a663-5deba830d1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d" Nov 03 02:10:08.558098 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:08.558064 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661ba085-9a33-4617-a663-5deba830d1fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dfbda7e-7h26d\" (UID: \"661ba085-9a33-4617-a663-5deba830d1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d" Nov 03 02:10:08.558300 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:08.558114 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661ba085-9a33-4617-a663-5deba830d1fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dfbda7e-7h26d\" (UID: \"661ba085-9a33-4617-a663-5deba830d1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d" Nov 03 02:10:08.558300 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:08.558167 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7c4j\" (UniqueName: \"kubernetes.io/projected/661ba085-9a33-4617-a663-5deba830d1fd-kube-api-access-d7c4j\") pod \"maas-api-revoke-sub-7dfbda7e-7h26d\" (UID: \"661ba085-9a33-4617-a663-5deba830d1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d" Nov 03 02:10:08.558833 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:08.558811 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661ba085-9a33-4617-a663-5deba830d1fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dfbda7e-7h26d\" (UID: \"661ba085-9a33-4617-a663-5deba830d1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d" Nov 03 02:10:08.560521 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:08.560493 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661ba085-9a33-4617-a663-5deba830d1fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dfbda7e-7h26d\" (UID: \"661ba085-9a33-4617-a663-5deba830d1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d" Nov 03 02:10:08.614995 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:08.614922 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7c4j\" (UniqueName: \"kubernetes.io/projected/661ba085-9a33-4617-a663-5deba830d1fd-kube-api-access-d7c4j\") pod \"maas-api-revoke-sub-7dfbda7e-7h26d\" (UID: \"661ba085-9a33-4617-a663-5deba830d1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d" Nov 03 02:10:08.641186 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:08.641163 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d" Nov 03 02:10:09.006110 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:10:09.006082 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661ba085_9a33_4617_a663_5deba830d1fd.slice/crio-c1072a5469bc61761901e9978722ea2a77777ab0f1cc5b497619052240b6f79d WatchSource:0}: Error finding container c1072a5469bc61761901e9978722ea2a77777ab0f1cc5b497619052240b6f79d: Status 404 returned error can't find the container with id c1072a5469bc61761901e9978722ea2a77777ab0f1cc5b497619052240b6f79d Nov 03 02:10:09.008352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:09.008328 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d"] Nov 03 02:10:09.432909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:09.432881 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:09.646970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:09.646935 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d" event={"ID":"661ba085-9a33-4617-a663-5deba830d1fd","Type":"ContainerStarted","Data":"72a3b169a664bba36d3d245a2102dd8aedcd3942b607980891940f0851c2e916"} Nov 03 02:10:09.647161 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:09.646995 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d" event={"ID":"661ba085-9a33-4617-a663-5deba830d1fd","Type":"ContainerStarted","Data":"c1072a5469bc61761901e9978722ea2a77777ab0f1cc5b497619052240b6f79d"} Nov 03 02:10:09.763603 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:09.763549 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d" podStartSLOduration=1.76353425 podStartE2EDuration="1.76353425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:10:09.761299901 +0000 UTC m=+1091.445409052" watchObservedRunningTime="2026-11-03 02:10:09.76353425 +0000 UTC m=+1091.447643405" Nov 03 02:10:10.655304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:10.655271 5463 generic.go:356] "Generic (PLEG): container finished" podID="661ba085-9a33-4617-a663-5deba830d1fd" containerID="72a3b169a664bba36d3d245a2102dd8aedcd3942b607980891940f0851c2e916" exitCode=0 Nov 03 02:10:10.655682 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:10.655351 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d" event={"ID":"661ba085-9a33-4617-a663-5deba830d1fd","Type":"ContainerDied","Data":"72a3b169a664bba36d3d245a2102dd8aedcd3942b607980891940f0851c2e916"} Nov 03 02:10:11.132514 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:11.132439 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:11.822472 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:11.822447 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d" Nov 03 02:10:11.998863 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:11.998782 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661ba085-9a33-4617-a663-5deba830d1fd-kube-api-access-d7c4j\" (UniqueName: \"kubernetes.io/projected/661ba085-9a33-4617-a663-5deba830d1fd-kube-api-access-d7c4j\") pod \"661ba085-9a33-4617-a663-5deba830d1fd\" (UID: \"661ba085-9a33-4617-a663-5deba830d1fd\") " Nov 03 02:10:11.999028 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:11.998907 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/661ba085-9a33-4617-a663-5deba830d1fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661ba085-9a33-4617-a663-5deba830d1fd-maas-api-xks-ca\") pod \"661ba085-9a33-4617-a663-5deba830d1fd\" (UID: \"661ba085-9a33-4617-a663-5deba830d1fd\") " Nov 03 02:10:11.999028 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:11.998940 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/661ba085-9a33-4617-a663-5deba830d1fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661ba085-9a33-4617-a663-5deba830d1fd-maas-api-openshift-ca\") pod \"661ba085-9a33-4617-a663-5deba830d1fd\" (UID: \"661ba085-9a33-4617-a663-5deba830d1fd\") " Nov 03 02:10:11.999366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:11.999342 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661ba085-9a33-4617-a663-5deba830d1fd-maas-api-openshift-ca" pod "661ba085-9a33-4617-a663-5deba830d1fd" (UID: "661ba085-9a33-4617-a663-5deba830d1fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:10:12.000808 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.000785 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661ba085-9a33-4617-a663-5deba830d1fd-kube-api-access-d7c4j" pod "661ba085-9a33-4617-a663-5deba830d1fd" (UID: "661ba085-9a33-4617-a663-5deba830d1fd"). InnerVolumeSpecName "kube-api-access-d7c4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:12.000910 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.000886 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/661ba085-9a33-4617-a663-5deba830d1fd-maas-api-xks-ca" pod "661ba085-9a33-4617-a663-5deba830d1fd" (UID: "661ba085-9a33-4617-a663-5deba830d1fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:10:12.100484 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.100461 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661ba085-9a33-4617-a663-5deba830d1fd-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:12.100484 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.100483 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661ba085-9a33-4617-a663-5deba830d1fd-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:12.100619 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.100492 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7c4j\" (UniqueName: \"kubernetes.io/projected/661ba085-9a33-4617-a663-5deba830d1fd-kube-api-access-d7c4j\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:12.426053 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.426020 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms"] Nov 03 02:10:12.426644 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.426627 5463 cpu_manager.go:414] "containerMap: removing container" podUID="661ba085-9a33-4617-a663-5deba830d1fd" containerName="revoke-keys" Nov 03 02:10:12.426644 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.426644 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661ba085-9a33-4617-a663-5deba830d1fd" containerName="revoke-keys" Nov 03 02:10:12.426779 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.426766 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="661ba085-9a33-4617-a663-5deba830d1fd" containerName="revoke-keys" Nov 03 02:10:12.430394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.430369 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.484157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.484127 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms"] Nov 03 02:10:12.605846 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.605812 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.605846 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.605849 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2w2k\" (UniqueName: \"kubernetes.io/projected/2394b438-fc9a-4747-8eb8-78e9f1524a1d-kube-api-access-g2w2k\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.606109 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.605874 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.606208 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.606167 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.606265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.606226 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.606306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.606290 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.606340 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.606321 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.606404 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.606342 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.606666 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.606644 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.671235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.671210 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d" Nov 03 02:10:12.671371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.671257 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d" event={"ID":"661ba085-9a33-4617-a663-5deba830d1fd","Type":"ContainerDied","Data":"c1072a5469bc61761901e9978722ea2a77777ab0f1cc5b497619052240b6f79d"} Nov 03 02:10:12.671371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.671313 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1072a5469bc61761901e9978722ea2a77777ab0f1cc5b497619052240b6f79d" Nov 03 02:10:12.708134 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.708070 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.708134 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.708103 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.708134 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.708125 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.708402 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.708159 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.708402 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.708221 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.708402 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.708352 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.708402 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.708397 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.708616 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.708421 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2w2k\" (UniqueName: \"kubernetes.io/projected/2394b438-fc9a-4747-8eb8-78e9f1524a1d-kube-api-access-g2w2k\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.708616 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.708462 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.708616 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.708516 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.708764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.708737 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.708834 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.708761 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.708834 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.708792 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.709094 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.709075 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.711051 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.711022 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.711144 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.711108 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.765237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.765180 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:12.765385 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:12.765278 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2w2k\" (UniqueName: \"kubernetes.io/projected/2394b438-fc9a-4747-8eb8-78e9f1524a1d-kube-api-access-g2w2k\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:13.008479 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:10:13.008395 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:10:13.045810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:13.045780 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:13.467573 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:10:13.467541 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2394b438_fc9a_4747_8eb8_78e9f1524a1d.slice/crio-a0da7de4be5a16aedc933891fc61a5a572749de08db3ddbc4132761e261b35ec WatchSource:0}: Error finding container a0da7de4be5a16aedc933891fc61a5a572749de08db3ddbc4132761e261b35ec: Status 404 returned error can't find the container with id a0da7de4be5a16aedc933891fc61a5a572749de08db3ddbc4132761e261b35ec Nov 03 02:10:13.470571 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:13.470544 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms"] Nov 03 02:10:13.470697 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:13.470657 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:10:13.470760 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:13.470744 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:10:13.470802 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:13.470788 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:10:13.682247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:13.682177 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" event={"ID":"2394b438-fc9a-4747-8eb8-78e9f1524a1d","Type":"ContainerStarted","Data":"4199f540700f2ed29d2a34f70e3dec7b0e78872698e6bb48e703bd78a790c940"} Nov 03 02:10:13.682423 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:13.682272 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" event={"ID":"2394b438-fc9a-4747-8eb8-78e9f1524a1d","Type":"ContainerStarted","Data":"a0da7de4be5a16aedc933891fc61a5a572749de08db3ddbc4132761e261b35ec"} Nov 03 02:10:13.798727 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:13.798627 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" podStartSLOduration=1.798613523 podStartE2EDuration="1.798613523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:10:13.797423207 +0000 UTC m=+1095.481532363" watchObservedRunningTime="2026-11-03 02:10:13.798613523 +0000 UTC m=+1095.482722678" Nov 03 02:10:14.046609 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:14.046579 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:14.048345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:14.048318 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 02:10:14.048494 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:14.048364 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" podUID="2394b438-fc9a-4747-8eb8-78e9f1524a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 02:10:15.046996 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.046966 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 02:10:15.047486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.047014 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" podUID="2394b438-fc9a-4747-8eb8-78e9f1524a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 02:10:15.073565 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.073530 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl"] Nov 03 02:10:15.077662 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.077641 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl" Nov 03 02:10:15.119575 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.119539 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl"] Nov 03 02:10:15.172955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.172568 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:10:15.172955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.172712 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:10:15.172955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.172756 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:10:15.172955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.172783 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:10:15.213212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.213163 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5"] Nov 03 02:10:15.217349 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.217331 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.239074 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.239045 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a5a567-bf98-411f-95bf-394a62dfab0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d002e070-d5pfl\" (UID: \"d9a5a567-bf98-411f-95bf-394a62dfab0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl" Nov 03 02:10:15.239286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.239265 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a5a567-bf98-411f-95bf-394a62dfab0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d002e070-d5pfl\" (UID: \"d9a5a567-bf98-411f-95bf-394a62dfab0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl" Nov 03 02:10:15.239393 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.239373 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcg54\" (UniqueName: \"kubernetes.io/projected/d9a5a567-bf98-411f-95bf-394a62dfab0d-kube-api-access-kcg54\") pod \"maas-api-revoke-sub-d002e070-d5pfl\" (UID: \"d9a5a567-bf98-411f-95bf-394a62dfab0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl" Nov 03 02:10:15.303068 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.302989 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5"] Nov 03 02:10:15.340386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.340348 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.340582 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.340407 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcg54\" (UniqueName: \"kubernetes.io/projected/d9a5a567-bf98-411f-95bf-394a62dfab0d-kube-api-access-kcg54\") pod \"maas-api-revoke-sub-d002e070-d5pfl\" (UID: \"d9a5a567-bf98-411f-95bf-394a62dfab0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl" Nov 03 02:10:15.340582 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.340505 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.340682 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.340657 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.340741 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.340706 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhs98\" (UniqueName: \"kubernetes.io/projected/731a025d-7302-4a22-8b43-70f96d8f05bb-kube-api-access-xhs98\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.340845 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.340821 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a5a567-bf98-411f-95bf-394a62dfab0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d002e070-d5pfl\" (UID: \"d9a5a567-bf98-411f-95bf-394a62dfab0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl" Nov 03 02:10:15.340931 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.340856 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.341066 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.340936 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.341066 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.340965 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.341186 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.341075 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.341186 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.341104 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a5a567-bf98-411f-95bf-394a62dfab0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d002e070-d5pfl\" (UID: \"d9a5a567-bf98-411f-95bf-394a62dfab0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl" Nov 03 02:10:15.341368 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.341330 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/731a025d-7302-4a22-8b43-70f96d8f05bb-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.341912 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.341887 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a5a567-bf98-411f-95bf-394a62dfab0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d002e070-d5pfl\" (UID: \"d9a5a567-bf98-411f-95bf-394a62dfab0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl" Nov 03 02:10:15.343766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.343746 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a5a567-bf98-411f-95bf-394a62dfab0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d002e070-d5pfl\" (UID: \"d9a5a567-bf98-411f-95bf-394a62dfab0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl" Nov 03 02:10:15.397239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.397207 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcg54\" (UniqueName: \"kubernetes.io/projected/d9a5a567-bf98-411f-95bf-394a62dfab0d-kube-api-access-kcg54\") pod \"maas-api-revoke-sub-d002e070-d5pfl\" (UID: \"d9a5a567-bf98-411f-95bf-394a62dfab0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl" Nov 03 02:10:15.442838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.442810 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.442838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.442841 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhs98\" (UniqueName: \"kubernetes.io/projected/731a025d-7302-4a22-8b43-70f96d8f05bb-kube-api-access-xhs98\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.443034 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.442973 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.443114 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.443064 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.443114 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.443094 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.443468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.443165 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.443468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.443260 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.443468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.443327 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/731a025d-7302-4a22-8b43-70f96d8f05bb-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.443468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.443386 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.443468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.443430 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.443793 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.443521 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.443793 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.443769 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.443899 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.443874 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.444278 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.444249 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/731a025d-7302-4a22-8b43-70f96d8f05bb-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.445506 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.445483 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.446462 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.446444 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.501876 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.501839 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.502115 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.502088 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhs98\" (UniqueName: \"kubernetes.io/projected/731a025d-7302-4a22-8b43-70f96d8f05bb-kube-api-access-xhs98\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.530829 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.530797 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:15.698214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.697762 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl" Nov 03 02:10:15.993048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:15.892810 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:16.066452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:16.066233 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 02:10:16.066452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:16.066295 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" podUID="2394b438-fc9a-4747-8eb8-78e9f1524a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 02:10:16.736851 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:16.736796 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:10:16.819552 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:16.819495 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:10:16.819869 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:16.819838 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:10:16.846529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:16.846488 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5"] Nov 03 02:10:16.939292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:16.933904 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:16.971234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:16.970566 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" event={"ID":"731a025d-7302-4a22-8b43-70f96d8f05bb","Type":"ContainerStarted","Data":"95fe22851627fae52ba1e7ff045574d9706aea39a2cc9fc96ab1654775f8a3e4"} Nov 03 02:10:17.060216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:17.058303 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 02:10:17.060216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:17.058367 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" podUID="2394b438-fc9a-4747-8eb8-78e9f1524a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 02:10:17.090211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:17.089452 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl"] Nov 03 02:10:17.317775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:17.317736 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:17.428215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:17.427445 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:17.502225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:17.500776 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:17.947215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:17.934279 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:17.947215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:17.934666 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:10:18.038969 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:18.038935 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl" event={"ID":"d9a5a567-bf98-411f-95bf-394a62dfab0d","Type":"ContainerStarted","Data":"1a3e4e42fc4071b31c87e101be01889db308976698bf97b6e58a427ebf2f01b7"} Nov 03 02:10:18.039261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:18.039237 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl" event={"ID":"d9a5a567-bf98-411f-95bf-394a62dfab0d","Type":"ContainerStarted","Data":"7a4e2f95a798aa4ee8f6e250955f4e42f60d5aa66882973ea47d6621ba8d9691"} Nov 03 02:10:18.052265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:18.052235 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 02:10:18.052514 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:18.052487 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" podUID="2394b438-fc9a-4747-8eb8-78e9f1524a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 02:10:18.077483 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:18.077445 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" event={"ID":"731a025d-7302-4a22-8b43-70f96d8f05bb","Type":"ContainerStarted","Data":"28fd5f6ba5091a131cb357b2f1635f94929e6ec12bcb32112d8adcff51847e90"} Nov 03 02:10:18.401271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:18.333757 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:18.401271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:18.372240 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:10:18.507246 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:18.476303 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 02:10:18.522237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:18.508645 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:18.555403 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:18.555328 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 02:10:18.555644 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:18.555613 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" podUID="731a025d-7302-4a22-8b43-70f96d8f05bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 02:10:18.785216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:18.767303 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" podStartSLOduration=3.767267519 podStartE2EDuration="3.767267519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:10:18.486627117 +0000 UTC m=+1100.170736270" watchObservedRunningTime="2026-11-03 02:10:18.767267519 +0000 UTC m=+1100.451376675" Nov 03 02:10:18.843047 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:18.839731 5463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:18.846250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:18.846222 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:10:18.846461 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:18.846445 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:10:18.862124 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:18.862099 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:18.986776 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:18.986675 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:19.050627 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:19.050593 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 02:10:19.060298 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:19.060258 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" podUID="2394b438-fc9a-4747-8eb8-78e9f1524a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 02:10:19.351214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:19.349518 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:19.412211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:19.411492 5463 generic.go:356] "Generic (PLEG): container finished" podID="d9a5a567-bf98-411f-95bf-394a62dfab0d" containerID="1a3e4e42fc4071b31c87e101be01889db308976698bf97b6e58a427ebf2f01b7" exitCode=0 Nov 03 02:10:19.449216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:19.413244 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl" event={"ID":"d9a5a567-bf98-411f-95bf-394a62dfab0d","Type":"ContainerDied","Data":"1a3e4e42fc4071b31c87e101be01889db308976698bf97b6e58a427ebf2f01b7"} Nov 03 02:10:19.449216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:19.444817 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:10:19.530609 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:19.487883 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 03 02:10:19.530609 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:19.489451 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:19.532176 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:19.532043 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 02:10:19.532176 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:19.532114 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" podUID="731a025d-7302-4a22-8b43-70f96d8f05bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 02:10:19.660183 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:19.660143 5463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:19.728334 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:19.728306 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:10:19.842309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:19.842269 5463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:19.939154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:19.938118 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:10:20.044536 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:20.044506 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:10:20.046377 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:20.046348 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 02:10:20.046612 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:20.046586 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" podUID="2394b438-fc9a-4747-8eb8-78e9f1524a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 02:10:20.531243 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:20.531207 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 02:10:20.532155 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:20.532125 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" podUID="731a025d-7302-4a22-8b43-70f96d8f05bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 02:10:20.899911 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:20.856597 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl" Nov 03 02:10:20.965550 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:20.965515 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:21.010214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:21.007735 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a5a567-bf98-411f-95bf-394a62dfab0d-kube-api-access-kcg54\" (UniqueName: \"kubernetes.io/projected/d9a5a567-bf98-411f-95bf-394a62dfab0d-kube-api-access-kcg54\") pod \"d9a5a567-bf98-411f-95bf-394a62dfab0d\" (UID: \"d9a5a567-bf98-411f-95bf-394a62dfab0d\") " Nov 03 02:10:21.010214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:21.007865 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9a5a567-bf98-411f-95bf-394a62dfab0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a5a567-bf98-411f-95bf-394a62dfab0d-maas-api-openshift-ca\") pod \"d9a5a567-bf98-411f-95bf-394a62dfab0d\" (UID: \"d9a5a567-bf98-411f-95bf-394a62dfab0d\") " Nov 03 02:10:21.010214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:21.007903 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9a5a567-bf98-411f-95bf-394a62dfab0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a5a567-bf98-411f-95bf-394a62dfab0d-maas-api-xks-ca\") pod \"d9a5a567-bf98-411f-95bf-394a62dfab0d\" (UID: \"d9a5a567-bf98-411f-95bf-394a62dfab0d\") " Nov 03 02:10:21.010214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:21.009580 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a5a567-bf98-411f-95bf-394a62dfab0d-maas-api-openshift-ca" pod "d9a5a567-bf98-411f-95bf-394a62dfab0d" (UID: "d9a5a567-bf98-411f-95bf-394a62dfab0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:10:21.028831 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:21.028789 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9a5a567-bf98-411f-95bf-394a62dfab0d-maas-api-xks-ca" pod "d9a5a567-bf98-411f-95bf-394a62dfab0d" (UID: "d9a5a567-bf98-411f-95bf-394a62dfab0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:10:21.032874 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:21.032844 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a5a567-bf98-411f-95bf-394a62dfab0d-kube-api-access-kcg54" pod "d9a5a567-bf98-411f-95bf-394a62dfab0d" (UID: "d9a5a567-bf98-411f-95bf-394a62dfab0d"). InnerVolumeSpecName "kube-api-access-kcg54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:21.046755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:21.046733 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 02:10:21.046859 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:21.046782 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" podUID="2394b438-fc9a-4747-8eb8-78e9f1524a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 02:10:21.124276 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:21.112239 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a5a567-bf98-411f-95bf-394a62dfab0d-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:21.124276 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:21.112275 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a5a567-bf98-411f-95bf-394a62dfab0d-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:21.124276 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:21.112290 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcg54\" (UniqueName: \"kubernetes.io/projected/d9a5a567-bf98-411f-95bf-394a62dfab0d-kube-api-access-kcg54\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:21.435071 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:21.434511 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl" Nov 03 02:10:21.435071 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:21.434541 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl" event={"ID":"d9a5a567-bf98-411f-95bf-394a62dfab0d","Type":"ContainerDied","Data":"7a4e2f95a798aa4ee8f6e250955f4e42f60d5aa66882973ea47d6621ba8d9691"} Nov 03 02:10:21.435071 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:21.434596 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a4e2f95a798aa4ee8f6e250955f4e42f60d5aa66882973ea47d6621ba8d9691" Nov 03 02:10:21.531516 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:21.531485 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 02:10:21.532008 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:21.531533 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" podUID="731a025d-7302-4a22-8b43-70f96d8f05bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 02:10:22.072216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:22.068287 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 02:10:22.072216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:22.068367 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" podUID="2394b438-fc9a-4747-8eb8-78e9f1524a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 02:10:22.086271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:22.086234 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568"] Nov 03 02:10:22.091400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:22.091372 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3783ebd-9w568"] Nov 03 02:10:22.445351 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:22.445317 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e3bab4f-46b5-4a0f-b16a-adc88af640c7" path="/var/lib/kubelet/pods/1e3bab4f-46b5-4a0f-b16a-adc88af640c7/volumes" Nov 03 02:10:22.531518 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:22.531481 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 02:10:22.531681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:22.531561 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" podUID="731a025d-7302-4a22-8b43-70f96d8f05bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 02:10:23.064738 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:23.064705 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:23.067022 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:23.067002 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:23.079509 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:23.079484 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:10:23.531568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:23.531537 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 02:10:23.531754 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:23.531583 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" podUID="731a025d-7302-4a22-8b43-70f96d8f05bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 02:10:24.531237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:24.531182 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 02:10:24.531405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:24.531257 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" podUID="731a025d-7302-4a22-8b43-70f96d8f05bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 02:10:24.745474 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:24.745443 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h"] Nov 03 02:10:24.746061 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:24.746045 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d9a5a567-bf98-411f-95bf-394a62dfab0d" containerName="revoke-keys" Nov 03 02:10:24.746106 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:24.746061 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a5a567-bf98-411f-95bf-394a62dfab0d" containerName="revoke-keys" Nov 03 02:10:24.746154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:24.746144 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a5a567-bf98-411f-95bf-394a62dfab0d" containerName="revoke-keys" Nov 03 02:10:24.772832 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:24.772799 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h" Nov 03 02:10:24.801630 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:24.801560 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h"] Nov 03 02:10:24.825630 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:24.825605 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:10:24.825771 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:24.825653 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:10:24.883806 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:24.883776 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd008509-2862-4597-bbd3-9321083f030a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cf2001e-mqw2h\" (UID: \"dd008509-2862-4597-bbd3-9321083f030a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h" Nov 03 02:10:24.884061 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:24.884044 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhrcr\" (UniqueName: \"kubernetes.io/projected/dd008509-2862-4597-bbd3-9321083f030a-kube-api-access-mhrcr\") pod \"maas-api-revoke-sub-4cf2001e-mqw2h\" (UID: \"dd008509-2862-4597-bbd3-9321083f030a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h" Nov 03 02:10:24.884270 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:24.884254 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd008509-2862-4597-bbd3-9321083f030a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cf2001e-mqw2h\" (UID: \"dd008509-2862-4597-bbd3-9321083f030a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h" Nov 03 02:10:24.893907 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:24.893883 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:10:24.894105 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:24.894085 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:10:24.986348 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:24.986311 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd008509-2862-4597-bbd3-9321083f030a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cf2001e-mqw2h\" (UID: \"dd008509-2862-4597-bbd3-9321083f030a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h" Nov 03 02:10:24.986348 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:24.986353 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrcr\" (UniqueName: \"kubernetes.io/projected/dd008509-2862-4597-bbd3-9321083f030a-kube-api-access-mhrcr\") pod \"maas-api-revoke-sub-4cf2001e-mqw2h\" (UID: \"dd008509-2862-4597-bbd3-9321083f030a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h" Nov 03 02:10:24.986630 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:24.986421 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd008509-2862-4597-bbd3-9321083f030a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cf2001e-mqw2h\" (UID: \"dd008509-2862-4597-bbd3-9321083f030a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h" Nov 03 02:10:24.987158 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:24.987131 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd008509-2862-4597-bbd3-9321083f030a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cf2001e-mqw2h\" (UID: \"dd008509-2862-4597-bbd3-9321083f030a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h" Nov 03 02:10:24.989142 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:24.989117 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd008509-2862-4597-bbd3-9321083f030a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cf2001e-mqw2h\" (UID: \"dd008509-2862-4597-bbd3-9321083f030a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h" Nov 03 02:10:25.043498 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:25.043465 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrcr\" (UniqueName: \"kubernetes.io/projected/dd008509-2862-4597-bbd3-9321083f030a-kube-api-access-mhrcr\") pod \"maas-api-revoke-sub-4cf2001e-mqw2h\" (UID: \"dd008509-2862-4597-bbd3-9321083f030a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h" Nov 03 02:10:25.089169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:25.089102 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h" Nov 03 02:10:25.461810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:25.461783 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h"] Nov 03 02:10:25.488780 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:25.488750 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h" event={"ID":"dd008509-2862-4597-bbd3-9321083f030a","Type":"ContainerStarted","Data":"63f6a8fe48ae0f71b18b9fca01baac16791edd168cbac5696f7c069f9421a803"} Nov 03 02:10:25.531157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:25.531121 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 02:10:25.531277 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:25.531179 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" podUID="731a025d-7302-4a22-8b43-70f96d8f05bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 02:10:25.727390 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:25.727317 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:26.497074 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:26.497041 5463 generic.go:356] "Generic (PLEG): container finished" podID="dd008509-2862-4597-bbd3-9321083f030a" containerID="25803946466bfa1e80d722d603b66215e117706891df48ceb7b357f6bdfd6bfc" exitCode=0 Nov 03 02:10:26.497505 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:26.497140 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h" event={"ID":"dd008509-2862-4597-bbd3-9321083f030a","Type":"ContainerDied","Data":"25803946466bfa1e80d722d603b66215e117706891df48ceb7b357f6bdfd6bfc"} Nov 03 02:10:26.535833 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:26.535803 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:26.536095 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:26.536064 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:26.536791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:26.536773 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:10:27.656646 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:27.656618 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h" Nov 03 02:10:27.720520 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:27.720494 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd008509-2862-4597-bbd3-9321083f030a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd008509-2862-4597-bbd3-9321083f030a-maas-api-xks-ca\") pod \"dd008509-2862-4597-bbd3-9321083f030a\" (UID: \"dd008509-2862-4597-bbd3-9321083f030a\") " Nov 03 02:10:27.720679 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:27.720634 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd008509-2862-4597-bbd3-9321083f030a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd008509-2862-4597-bbd3-9321083f030a-maas-api-openshift-ca\") pod \"dd008509-2862-4597-bbd3-9321083f030a\" (UID: \"dd008509-2862-4597-bbd3-9321083f030a\") " Nov 03 02:10:27.720997 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:27.720975 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd008509-2862-4597-bbd3-9321083f030a-maas-api-openshift-ca" pod "dd008509-2862-4597-bbd3-9321083f030a" (UID: "dd008509-2862-4597-bbd3-9321083f030a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:10:27.721102 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:27.721088 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd008509-2862-4597-bbd3-9321083f030a-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:27.722462 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:27.722437 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd008509-2862-4597-bbd3-9321083f030a-maas-api-xks-ca" pod "dd008509-2862-4597-bbd3-9321083f030a" (UID: "dd008509-2862-4597-bbd3-9321083f030a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:10:27.822343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:27.822268 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd008509-2862-4597-bbd3-9321083f030a-kube-api-access-mhrcr\" (UniqueName: \"kubernetes.io/projected/dd008509-2862-4597-bbd3-9321083f030a-kube-api-access-mhrcr\") pod \"dd008509-2862-4597-bbd3-9321083f030a\" (UID: \"dd008509-2862-4597-bbd3-9321083f030a\") " Nov 03 02:10:27.822919 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:27.822888 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd008509-2862-4597-bbd3-9321083f030a-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:27.824349 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:27.824328 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd008509-2862-4597-bbd3-9321083f030a-kube-api-access-mhrcr" pod "dd008509-2862-4597-bbd3-9321083f030a" (UID: "dd008509-2862-4597-bbd3-9321083f030a"). InnerVolumeSpecName "kube-api-access-mhrcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:27.926456 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:27.924994 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhrcr\" (UniqueName: \"kubernetes.io/projected/dd008509-2862-4597-bbd3-9321083f030a-kube-api-access-mhrcr\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:28.513108 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:28.513083 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h" Nov 03 02:10:28.513292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:28.513119 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h" event={"ID":"dd008509-2862-4597-bbd3-9321083f030a","Type":"ContainerDied","Data":"63f6a8fe48ae0f71b18b9fca01baac16791edd168cbac5696f7c069f9421a803"} Nov 03 02:10:28.513292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:28.513168 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63f6a8fe48ae0f71b18b9fca01baac16791edd168cbac5696f7c069f9421a803" Nov 03 02:10:30.443253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.443220 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:30.554452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.554410 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm"] Nov 03 02:10:30.555137 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.555114 5463 cpu_manager.go:414] "containerMap: removing container" podUID="dd008509-2862-4597-bbd3-9321083f030a" containerName="revoke-keys" Nov 03 02:10:30.555137 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.555135 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd008509-2862-4597-bbd3-9321083f030a" containerName="revoke-keys" Nov 03 02:10:30.555347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.555312 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd008509-2862-4597-bbd3-9321083f030a" containerName="revoke-keys" Nov 03 02:10:30.559150 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.559128 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm" Nov 03 02:10:30.611977 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.611936 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:10:30.658429 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.658391 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wjqf\" (UniqueName: \"kubernetes.io/projected/1137541f-61df-4c91-a030-d4c6c8e30778-kube-api-access-8wjqf\") pod \"maas-api-revoke-sub-c4ca6ac6-hqgwm\" (UID: \"1137541f-61df-4c91-a030-d4c6c8e30778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm" Nov 03 02:10:30.658620 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.658453 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1137541f-61df-4c91-a030-d4c6c8e30778-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4ca6ac6-hqgwm\" (UID: \"1137541f-61df-4c91-a030-d4c6c8e30778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm" Nov 03 02:10:30.658620 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.658612 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1137541f-61df-4c91-a030-d4c6c8e30778-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4ca6ac6-hqgwm\" (UID: \"1137541f-61df-4c91-a030-d4c6c8e30778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm" Nov 03 02:10:30.662964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.662550 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:10:30.662964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.662551 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:10:30.662964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.662663 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:10:30.667091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.667062 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm"] Nov 03 02:10:30.760187 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.760115 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjqf\" (UniqueName: \"kubernetes.io/projected/1137541f-61df-4c91-a030-d4c6c8e30778-kube-api-access-8wjqf\") pod \"maas-api-revoke-sub-c4ca6ac6-hqgwm\" (UID: \"1137541f-61df-4c91-a030-d4c6c8e30778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm" Nov 03 02:10:30.760187 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.760155 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1137541f-61df-4c91-a030-d4c6c8e30778-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4ca6ac6-hqgwm\" (UID: \"1137541f-61df-4c91-a030-d4c6c8e30778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm" Nov 03 02:10:30.760427 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.760277 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1137541f-61df-4c91-a030-d4c6c8e30778-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4ca6ac6-hqgwm\" (UID: \"1137541f-61df-4c91-a030-d4c6c8e30778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm" Nov 03 02:10:30.760939 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.760915 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1137541f-61df-4c91-a030-d4c6c8e30778-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4ca6ac6-hqgwm\" (UID: \"1137541f-61df-4c91-a030-d4c6c8e30778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm" Nov 03 02:10:30.762636 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.762606 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1137541f-61df-4c91-a030-d4c6c8e30778-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4ca6ac6-hqgwm\" (UID: \"1137541f-61df-4c91-a030-d4c6c8e30778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm" Nov 03 02:10:30.817522 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.817490 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wjqf\" (UniqueName: \"kubernetes.io/projected/1137541f-61df-4c91-a030-d4c6c8e30778-kube-api-access-8wjqf\") pod \"maas-api-revoke-sub-c4ca6ac6-hqgwm\" (UID: \"1137541f-61df-4c91-a030-d4c6c8e30778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm" Nov 03 02:10:30.873073 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:30.873042 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm" Nov 03 02:10:31.239371 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:10:31.239336 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1137541f_61df_4c91_a030_d4c6c8e30778.slice/crio-a8a0cdd3c46bde2e452ab1cbce7eda32f60c4588b0dc304b8d18f76452faebd1 WatchSource:0}: Error finding container a8a0cdd3c46bde2e452ab1cbce7eda32f60c4588b0dc304b8d18f76452faebd1: Status 404 returned error can't find the container with id a8a0cdd3c46bde2e452ab1cbce7eda32f60c4588b0dc304b8d18f76452faebd1 Nov 03 02:10:31.243741 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.243722 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm"] Nov 03 02:10:31.540519 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.540486 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm" event={"ID":"1137541f-61df-4c91-a030-d4c6c8e30778","Type":"ContainerStarted","Data":"a8a0cdd3c46bde2e452ab1cbce7eda32f60c4588b0dc304b8d18f76452faebd1"} Nov 03 02:10:31.709494 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.709462 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8"] Nov 03 02:10:31.713884 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.713857 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.776769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.776745 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.776885 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.776783 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.776936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.776877 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rck85\" (UniqueName: \"kubernetes.io/projected/cc2f2fb4-5a6f-4778-90e5-9a857204f663-kube-api-access-rck85\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.776936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.776913 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.777011 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.777000 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.777046 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.777036 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.777080 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.777053 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.777080 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.777069 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.777149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.777084 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.818735 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.818699 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8"] Nov 03 02:10:31.878151 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.878119 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.878326 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.878174 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.878326 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.878245 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rck85\" (UniqueName: \"kubernetes.io/projected/cc2f2fb4-5a6f-4778-90e5-9a857204f663-kube-api-access-rck85\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.878326 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.878277 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.878517 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.878377 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.878517 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.878431 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.878517 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.878458 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.878517 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.878488 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.878517 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.878514 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.878770 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.878723 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.879153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.879103 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.879293 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.879174 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.879293 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.879238 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.879580 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.879552 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.881141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.881078 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.881326 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.881306 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.935712 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.935676 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:31.935886 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:31.935862 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rck85\" (UniqueName: \"kubernetes.io/projected/cc2f2fb4-5a6f-4778-90e5-9a857204f663-kube-api-access-rck85\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:32.028366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.028293 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:32.452027 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:10:32.451995 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2f2fb4_5a6f_4778_90e5_9a857204f663.slice/crio-844dcac3ae89c7bcd82145c75d87a37bac9c537365e622c26a8404a0533267c6 WatchSource:0}: Error finding container 844dcac3ae89c7bcd82145c75d87a37bac9c537365e622c26a8404a0533267c6: Status 404 returned error can't find the container with id 844dcac3ae89c7bcd82145c75d87a37bac9c537365e622c26a8404a0533267c6 Nov 03 02:10:32.454432 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.454411 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8"] Nov 03 02:10:32.455126 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.455092 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:10:32.455237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.455154 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:10:32.455237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.455184 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:10:32.552005 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.551971 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" event={"ID":"cc2f2fb4-5a6f-4778-90e5-9a857204f663","Type":"ContainerStarted","Data":"35464c1f7916006f4cedd3f0b5ba10a46959f8a5e4b006e1097fe0ee2a577f0e"} Nov 03 02:10:32.552537 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.552038 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" event={"ID":"cc2f2fb4-5a6f-4778-90e5-9a857204f663","Type":"ContainerStarted","Data":"844dcac3ae89c7bcd82145c75d87a37bac9c537365e622c26a8404a0533267c6"} Nov 03 02:10:32.553993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.553967 5463 generic.go:356] "Generic (PLEG): container finished" podID="1137541f-61df-4c91-a030-d4c6c8e30778" containerID="5c780aca00658e0539d6ada1850baddac90f69459fcbab862f2e1ab311ec8f9a" exitCode=0 Nov 03 02:10:32.554099 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.554030 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm" event={"ID":"1137541f-61df-4c91-a030-d4c6c8e30778","Type":"ContainerDied","Data":"5c780aca00658e0539d6ada1850baddac90f69459fcbab862f2e1ab311ec8f9a"} Nov 03 02:10:32.668964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.668904 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" podStartSLOduration=1.66888544 podStartE2EDuration="1.66888544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:10:32.667895606 +0000 UTC m=+1114.352004764" watchObservedRunningTime="2026-11-03 02:10:32.66888544 +0000 UTC m=+1114.352994601" Nov 03 02:10:32.852070 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.852032 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5"] Nov 03 02:10:32.856283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.856260 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.892497 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.892463 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.892684 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.892649 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.892684 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.892679 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stnqc\" (UniqueName: \"kubernetes.io/projected/d52f99a6-11e1-4a01-b565-4d3730765a20-kube-api-access-stnqc\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.892807 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.892740 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.892807 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.892778 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.892909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.892867 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.892909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.892897 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.893015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.892925 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52f99a6-11e1-4a01-b565-4d3730765a20-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.893015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.892983 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.915673 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.915639 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5"] Nov 03 02:10:32.994821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.994739 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.995225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.995182 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.995347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.995289 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.995347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.995317 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stnqc\" (UniqueName: \"kubernetes.io/projected/d52f99a6-11e1-4a01-b565-4d3730765a20-kube-api-access-stnqc\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.995460 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.995382 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.995460 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.995426 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.995560 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.995518 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.995560 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.995546 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.995752 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.995577 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52f99a6-11e1-4a01-b565-4d3730765a20-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.995752 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.995632 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.995953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.995937 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.996090 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.996067 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.996685 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.996659 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.996984 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.996958 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52f99a6-11e1-4a01-b565-4d3730765a20-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.998874 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.998849 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:32.998984 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:32.998971 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:33.029055 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:33.029036 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:33.030740 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:33.030718 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 02:10:33.030836 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:33.030765 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" podUID="cc2f2fb4-5a6f-4778-90e5-9a857204f663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 02:10:33.054712 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:33.054651 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:33.054843 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:33.054760 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stnqc\" (UniqueName: \"kubernetes.io/projected/d52f99a6-11e1-4a01-b565-4d3730765a20-kube-api-access-stnqc\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:33.175039 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:33.175005 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:33.601260 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:10:33.601104 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52f99a6_11e1_4a01_b565_4d3730765a20.slice/crio-71fd01aaf34fa90c2b0cc37cc702fcf08ccdf96f59edeb7c5790acd752b4bce1 WatchSource:0}: Error finding container 71fd01aaf34fa90c2b0cc37cc702fcf08ccdf96f59edeb7c5790acd752b4bce1: Status 404 returned error can't find the container with id 71fd01aaf34fa90c2b0cc37cc702fcf08ccdf96f59edeb7c5790acd752b4bce1 Nov 03 02:10:33.601670 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:33.601486 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5"] Nov 03 02:10:33.602812 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:33.602774 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:10:33.602914 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:33.602846 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:10:33.602914 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:33.602888 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:10:33.827769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:33.827745 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm" Nov 03 02:10:33.908013 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:33.907985 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1137541f-61df-4c91-a030-d4c6c8e30778-kube-api-access-8wjqf\" (UniqueName: \"kubernetes.io/projected/1137541f-61df-4c91-a030-d4c6c8e30778-kube-api-access-8wjqf\") pod \"1137541f-61df-4c91-a030-d4c6c8e30778\" (UID: \"1137541f-61df-4c91-a030-d4c6c8e30778\") " Nov 03 02:10:33.908177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:33.908028 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1137541f-61df-4c91-a030-d4c6c8e30778-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1137541f-61df-4c91-a030-d4c6c8e30778-maas-api-openshift-ca\") pod \"1137541f-61df-4c91-a030-d4c6c8e30778\" (UID: \"1137541f-61df-4c91-a030-d4c6c8e30778\") " Nov 03 02:10:33.908177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:33.908054 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1137541f-61df-4c91-a030-d4c6c8e30778-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1137541f-61df-4c91-a030-d4c6c8e30778-maas-api-xks-ca\") pod \"1137541f-61df-4c91-a030-d4c6c8e30778\" (UID: \"1137541f-61df-4c91-a030-d4c6c8e30778\") " Nov 03 02:10:33.908986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:33.908963 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1137541f-61df-4c91-a030-d4c6c8e30778-maas-api-openshift-ca" pod "1137541f-61df-4c91-a030-d4c6c8e30778" (UID: "1137541f-61df-4c91-a030-d4c6c8e30778"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:10:33.910519 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:33.910493 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1137541f-61df-4c91-a030-d4c6c8e30778-kube-api-access-8wjqf" pod "1137541f-61df-4c91-a030-d4c6c8e30778" (UID: "1137541f-61df-4c91-a030-d4c6c8e30778"). InnerVolumeSpecName "kube-api-access-8wjqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:33.910627 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:33.910549 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1137541f-61df-4c91-a030-d4c6c8e30778-maas-api-xks-ca" pod "1137541f-61df-4c91-a030-d4c6c8e30778" (UID: "1137541f-61df-4c91-a030-d4c6c8e30778"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:10:34.009527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:34.009470 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wjqf\" (UniqueName: \"kubernetes.io/projected/1137541f-61df-4c91-a030-d4c6c8e30778-kube-api-access-8wjqf\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:34.009527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:34.009497 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1137541f-61df-4c91-a030-d4c6c8e30778-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:34.009527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:34.009506 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1137541f-61df-4c91-a030-d4c6c8e30778-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:34.029145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:34.029117 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 02:10:34.029296 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:34.029164 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" podUID="cc2f2fb4-5a6f-4778-90e5-9a857204f663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 02:10:34.614831 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:34.614625 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm" event={"ID":"1137541f-61df-4c91-a030-d4c6c8e30778","Type":"ContainerDied","Data":"a8a0cdd3c46bde2e452ab1cbce7eda32f60c4588b0dc304b8d18f76452faebd1"} Nov 03 02:10:34.614831 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:34.614673 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8a0cdd3c46bde2e452ab1cbce7eda32f60c4588b0dc304b8d18f76452faebd1" Nov 03 02:10:34.614831 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:34.614728 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm" Nov 03 02:10:34.617318 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:34.617284 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" event={"ID":"d52f99a6-11e1-4a01-b565-4d3730765a20","Type":"ContainerStarted","Data":"5eb2c2aec494e8610c7272d0be177ed492906458ed5049cae3a951f471f177b5"} Nov 03 02:10:34.617454 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:34.617339 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" event={"ID":"d52f99a6-11e1-4a01-b565-4d3730765a20","Type":"ContainerStarted","Data":"71fd01aaf34fa90c2b0cc37cc702fcf08ccdf96f59edeb7c5790acd752b4bce1"} Nov 03 02:10:34.738429 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:34.738368 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" podStartSLOduration=2.738353577 podStartE2EDuration="2.738353577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:10:34.734491561 +0000 UTC m=+1116.418600716" watchObservedRunningTime="2026-11-03 02:10:34.738353577 +0000 UTC m=+1116.422462732" Nov 03 02:10:35.029431 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:35.029405 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 02:10:35.029675 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:35.029648 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" podUID="cc2f2fb4-5a6f-4778-90e5-9a857204f663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 02:10:35.181216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:35.177507 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:35.181216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:35.179689 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 02:10:35.181216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:35.179879 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" podUID="d52f99a6-11e1-4a01-b565-4d3730765a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 02:10:35.299265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:35.298562 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:36.028649 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:36.028610 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 02:10:36.029049 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:36.028672 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" podUID="cc2f2fb4-5a6f-4778-90e5-9a857204f663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 02:10:36.177567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:36.177530 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 02:10:36.177867 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:36.177837 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" podUID="d52f99a6-11e1-4a01-b565-4d3730765a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 02:10:36.469216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:36.465984 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:37.028889 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:37.028858 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 02:10:37.029297 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:37.028905 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" podUID="cc2f2fb4-5a6f-4778-90e5-9a857204f663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 02:10:37.175584 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:37.175545 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 02:10:37.175850 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:37.175820 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" podUID="d52f99a6-11e1-4a01-b565-4d3730765a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 02:10:37.636576 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:37.636538 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd"] Nov 03 02:10:37.637815 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:37.637790 5463 cpu_manager.go:414] "containerMap: removing container" podUID="1137541f-61df-4c91-a030-d4c6c8e30778" containerName="revoke-keys" Nov 03 02:10:37.637967 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:37.637952 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1137541f-61df-4c91-a030-d4c6c8e30778" containerName="revoke-keys" Nov 03 02:10:37.638276 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:37.638259 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="1137541f-61df-4c91-a030-d4c6c8e30778" containerName="revoke-keys" Nov 03 02:10:37.672226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:37.670685 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd" Nov 03 02:10:37.769211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:37.739780 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:10:37.769211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:37.739873 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:10:37.769211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:37.739957 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:10:37.793483 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:37.792769 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:10:38.237216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:38.173068 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 02:10:38.238245 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:38.173148 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" podUID="cc2f2fb4-5a6f-4778-90e5-9a857204f663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 02:10:38.259791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:38.259754 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4233154b-1891-4123-8c65-c5e910c1ab9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab7db544-56wpd\" (UID: \"4233154b-1891-4123-8c65-c5e910c1ab9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd" Nov 03 02:10:38.343657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:38.325210 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4233154b-1891-4123-8c65-c5e910c1ab9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab7db544-56wpd\" (UID: \"4233154b-1891-4123-8c65-c5e910c1ab9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd" Nov 03 02:10:38.343657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:38.325424 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrbz4\" (UniqueName: \"kubernetes.io/projected/4233154b-1891-4123-8c65-c5e910c1ab9a-kube-api-access-xrbz4\") pod \"maas-api-revoke-sub-ab7db544-56wpd\" (UID: \"4233154b-1891-4123-8c65-c5e910c1ab9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd" Nov 03 02:10:38.378215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:38.363761 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 02:10:38.378215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:38.363828 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" podUID="d52f99a6-11e1-4a01-b565-4d3730765a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 02:10:38.378215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:38.370334 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd"] Nov 03 02:10:38.395667 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:38.395344 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:38.434905 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:38.433057 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4233154b-1891-4123-8c65-c5e910c1ab9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab7db544-56wpd\" (UID: \"4233154b-1891-4123-8c65-c5e910c1ab9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd" Nov 03 02:10:38.434905 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:38.433111 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4233154b-1891-4123-8c65-c5e910c1ab9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab7db544-56wpd\" (UID: \"4233154b-1891-4123-8c65-c5e910c1ab9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd" Nov 03 02:10:38.434905 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:38.433287 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrbz4\" (UniqueName: \"kubernetes.io/projected/4233154b-1891-4123-8c65-c5e910c1ab9a-kube-api-access-xrbz4\") pod \"maas-api-revoke-sub-ab7db544-56wpd\" (UID: \"4233154b-1891-4123-8c65-c5e910c1ab9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd" Nov 03 02:10:38.434905 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:38.434503 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4233154b-1891-4123-8c65-c5e910c1ab9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab7db544-56wpd\" (UID: \"4233154b-1891-4123-8c65-c5e910c1ab9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd" Nov 03 02:10:38.473233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:38.465686 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4233154b-1891-4123-8c65-c5e910c1ab9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab7db544-56wpd\" (UID: \"4233154b-1891-4123-8c65-c5e910c1ab9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd" Nov 03 02:10:38.499225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:38.499055 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:38.524245 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:38.524173 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8"] Nov 03 02:10:38.624233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:38.616395 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb5a7e61-vvkd8"] Nov 03 02:10:38.714940 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:38.714904 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrbz4\" (UniqueName: \"kubernetes.io/projected/4233154b-1891-4123-8c65-c5e910c1ab9a-kube-api-access-xrbz4\") pod \"maas-api-revoke-sub-ab7db544-56wpd\" (UID: \"4233154b-1891-4123-8c65-c5e910c1ab9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd" Nov 03 02:10:38.728217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:38.715309 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd" Nov 03 02:10:39.012273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:39.001710 5463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:39.058468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:39.057400 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:39.058468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:39.058345 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 02:10:39.058468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:39.058392 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" podUID="cc2f2fb4-5a6f-4778-90e5-9a857204f663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 02:10:39.201230 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:39.177705 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 02:10:39.201230 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:39.177781 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" podUID="d52f99a6-11e1-4a01-b565-4d3730765a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 02:10:39.339324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:39.318299 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:39.349225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:39.346253 5463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:39.375227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:39.366267 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:39.405228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:39.401688 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 03 02:10:39.405228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:39.402073 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:10:39.509226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:39.505274 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:39.509226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:39.505718 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:10:39.891277 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:39.851526 5463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:40.135216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:40.064321 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:40.135216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:40.090037 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 02:10:40.135216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:40.090094 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" podUID="cc2f2fb4-5a6f-4778-90e5-9a857204f663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 02:10:40.135216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:40.098397 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:10:40.193215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:40.182266 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 02:10:40.193215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:40.182333 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" podUID="d52f99a6-11e1-4a01-b565-4d3730765a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 02:10:40.322371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:40.322321 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:40.342259 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:40.323336 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:10:40.393216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:40.377355 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:40.393216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:40.379391 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:10:40.413937 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:40.413882 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:40.463300 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:40.459338 5463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Nov 03 02:10:40.544527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:40.525325 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:40.825272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:40.787409 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31abc5f5-336f-4f7c-ba26-380a23e8ab9e" path="/var/lib/kubelet/pods/31abc5f5-336f-4f7c-ba26-380a23e8ab9e/volumes" Nov 03 02:10:40.866233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:40.828354 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:10:40.937983 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:40.937946 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" podUID="13e969fe-f189-40f2-b929-e462f6d4b86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:41.073601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:41.071716 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd"] Nov 03 02:10:41.073601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:41.072912 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 03 02:10:41.115304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:41.078710 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 02:10:41.115304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:41.078762 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" podUID="cc2f2fb4-5a6f-4778-90e5-9a857204f663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 02:10:41.115304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:41.079112 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:41.249218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:41.183986 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 02:10:41.269332 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:41.269278 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" podUID="d52f99a6-11e1-4a01-b565-4d3730765a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 02:10:41.389342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:41.328463 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:41.389342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:41.381264 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:41.423336 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:41.423282 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:41.536350 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:41.536302 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:42.007226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:42.004425 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:10:42.007226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:42.004534 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" podUID="8d2c4397-7a8c-41a5-9140-2079f97bd460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:42.055872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:42.041065 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 02:10:42.172915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:42.086517 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:42.197726 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:42.172869 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" podUID="cc2f2fb4-5a6f-4778-90e5-9a857204f663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 02:10:42.198372 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:42.198346 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 02:10:42.198525 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:42.198500 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" podUID="d52f99a6-11e1-4a01-b565-4d3730765a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 02:10:42.201489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:42.201468 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:10:42.238251 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:42.232296 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd" event={"ID":"4233154b-1891-4123-8c65-c5e910c1ab9a","Type":"ContainerStarted","Data":"f813b120d921b04523a042d2e626ffee31630e7dff7b8b807606afd65d7ecf80"} Nov 03 02:10:42.238251 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:42.232382 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd" event={"ID":"4233154b-1891-4123-8c65-c5e910c1ab9a","Type":"ContainerStarted","Data":"8223cbfd55658f0aaed53bfaf7fa68232f13023786fba12c6f7b2711c72350db"} Nov 03 02:10:42.430321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:42.424321 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:42.430321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:42.424673 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:42.444359 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:42.437516 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:42.496118 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:42.495852 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd" podStartSLOduration=5.495832687 podStartE2EDuration="5.495832687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:10:42.47872671 +0000 UTC m=+1124.162835866" watchObservedRunningTime="2026-11-03 02:10:42.495832687 +0000 UTC m=+1124.179941843" Nov 03 02:10:42.524380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:42.524351 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:10:42.541080 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:42.538567 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:10:42.541080 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:42.538624 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:10:42.555565 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:42.555503 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:10:43.029346 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:43.028833 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 02:10:43.029346 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:43.028888 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" podUID="cc2f2fb4-5a6f-4778-90e5-9a857204f663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 02:10:43.031226 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:10:43.031128 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:10:43.175926 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:43.175892 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 02:10:43.176097 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:43.175958 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" podUID="d52f99a6-11e1-4a01-b565-4d3730765a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 02:10:43.250151 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:43.250110 5463 generic.go:356] "Generic (PLEG): container finished" podID="4233154b-1891-4123-8c65-c5e910c1ab9a" containerID="f813b120d921b04523a042d2e626ffee31630e7dff7b8b807606afd65d7ecf80" exitCode=0 Nov 03 02:10:43.250534 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:43.250502 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd" event={"ID":"4233154b-1891-4123-8c65-c5e910c1ab9a","Type":"ContainerDied","Data":"f813b120d921b04523a042d2e626ffee31630e7dff7b8b807606afd65d7ecf80"} Nov 03 02:10:44.053246 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:44.053120 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf"] Nov 03 02:10:44.103904 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:44.103864 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46494c32-j9bxf"] Nov 03 02:10:44.475512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:44.475478 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6320d42b-af84-46aa-b036-812a3ca9c786" path="/var/lib/kubelet/pods/6320d42b-af84-46aa-b036-812a3ca9c786/volumes" Nov 03 02:10:44.536911 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:44.536875 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:44.538302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:44.538271 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:44.540021 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:44.539978 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:10:44.544405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:44.544383 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd" Nov 03 02:10:44.626034 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:44.625929 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4233154b-1891-4123-8c65-c5e910c1ab9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4233154b-1891-4123-8c65-c5e910c1ab9a-maas-api-xks-ca\") pod \"4233154b-1891-4123-8c65-c5e910c1ab9a\" (UID: \"4233154b-1891-4123-8c65-c5e910c1ab9a\") " Nov 03 02:10:44.626226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:44.626089 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4233154b-1891-4123-8c65-c5e910c1ab9a-kube-api-access-xrbz4\" (UniqueName: \"kubernetes.io/projected/4233154b-1891-4123-8c65-c5e910c1ab9a-kube-api-access-xrbz4\") pod \"4233154b-1891-4123-8c65-c5e910c1ab9a\" (UID: \"4233154b-1891-4123-8c65-c5e910c1ab9a\") " Nov 03 02:10:44.626226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:44.626161 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4233154b-1891-4123-8c65-c5e910c1ab9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4233154b-1891-4123-8c65-c5e910c1ab9a-maas-api-openshift-ca\") pod \"4233154b-1891-4123-8c65-c5e910c1ab9a\" (UID: \"4233154b-1891-4123-8c65-c5e910c1ab9a\") " Nov 03 02:10:44.626683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:44.626639 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4233154b-1891-4123-8c65-c5e910c1ab9a-maas-api-openshift-ca" pod "4233154b-1891-4123-8c65-c5e910c1ab9a" (UID: "4233154b-1891-4123-8c65-c5e910c1ab9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:10:44.629051 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:44.629019 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4233154b-1891-4123-8c65-c5e910c1ab9a-kube-api-access-xrbz4" pod "4233154b-1891-4123-8c65-c5e910c1ab9a" (UID: "4233154b-1891-4123-8c65-c5e910c1ab9a"). InnerVolumeSpecName "kube-api-access-xrbz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:44.629765 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:44.629741 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4233154b-1891-4123-8c65-c5e910c1ab9a-maas-api-xks-ca" pod "4233154b-1891-4123-8c65-c5e910c1ab9a" (UID: "4233154b-1891-4123-8c65-c5e910c1ab9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:10:44.727655 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:44.727618 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4233154b-1891-4123-8c65-c5e910c1ab9a-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:44.727881 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:44.727863 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4233154b-1891-4123-8c65-c5e910c1ab9a-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:44.727983 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:44.727970 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrbz4\" (UniqueName: \"kubernetes.io/projected/4233154b-1891-4123-8c65-c5e910c1ab9a-kube-api-access-xrbz4\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:45.030174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.030120 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:10:45.030585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.030553 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" podUID="cc2f2fb4-5a6f-4778-90e5-9a857204f663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:45.047151 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.047122 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:45.048245 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.048223 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:45.051278 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.051244 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:10:45.283528 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.283447 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd" Nov 03 02:10:45.290562 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.289460 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd" event={"ID":"4233154b-1891-4123-8c65-c5e910c1ab9a","Type":"ContainerDied","Data":"8223cbfd55658f0aaed53bfaf7fa68232f13023786fba12c6f7b2711c72350db"} Nov 03 02:10:45.290562 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.289515 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8223cbfd55658f0aaed53bfaf7fa68232f13023786fba12c6f7b2711c72350db" Nov 03 02:10:45.504253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.501523 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts"] Nov 03 02:10:45.504253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.502525 5463 cpu_manager.go:414] "containerMap: removing container" podUID="4233154b-1891-4123-8c65-c5e910c1ab9a" containerName="revoke-keys" Nov 03 02:10:45.504253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.502546 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4233154b-1891-4123-8c65-c5e910c1ab9a" containerName="revoke-keys" Nov 03 02:10:45.504253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.502722 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="4233154b-1891-4123-8c65-c5e910c1ab9a" containerName="revoke-keys" Nov 03 02:10:45.509233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.506991 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts" Nov 03 02:10:45.545284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.543689 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncmtj\" (UniqueName: \"kubernetes.io/projected/06609e21-7e29-454a-a6ad-4dced184ed62-kube-api-access-ncmtj\") pod \"maas-api-revoke-sub-7eda9adc-cntts\" (UID: \"06609e21-7e29-454a-a6ad-4dced184ed62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts" Nov 03 02:10:45.545284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.543969 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06609e21-7e29-454a-a6ad-4dced184ed62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eda9adc-cntts\" (UID: \"06609e21-7e29-454a-a6ad-4dced184ed62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts" Nov 03 02:10:45.545284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.544013 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06609e21-7e29-454a-a6ad-4dced184ed62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eda9adc-cntts\" (UID: \"06609e21-7e29-454a-a6ad-4dced184ed62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts" Nov 03 02:10:45.570097 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.570065 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:10:45.571112 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.570985 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:10:45.571495 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.571476 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:10:45.571717 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.571700 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:10:45.574029 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.573995 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts"] Nov 03 02:10:45.645437 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.645407 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06609e21-7e29-454a-a6ad-4dced184ed62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eda9adc-cntts\" (UID: \"06609e21-7e29-454a-a6ad-4dced184ed62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts" Nov 03 02:10:45.645603 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.645444 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06609e21-7e29-454a-a6ad-4dced184ed62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eda9adc-cntts\" (UID: \"06609e21-7e29-454a-a6ad-4dced184ed62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts" Nov 03 02:10:45.645657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.645605 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmtj\" (UniqueName: \"kubernetes.io/projected/06609e21-7e29-454a-a6ad-4dced184ed62-kube-api-access-ncmtj\") pod \"maas-api-revoke-sub-7eda9adc-cntts\" (UID: \"06609e21-7e29-454a-a6ad-4dced184ed62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts" Nov 03 02:10:45.646440 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.646267 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06609e21-7e29-454a-a6ad-4dced184ed62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eda9adc-cntts\" (UID: \"06609e21-7e29-454a-a6ad-4dced184ed62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts" Nov 03 02:10:45.648570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.648433 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06609e21-7e29-454a-a6ad-4dced184ed62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eda9adc-cntts\" (UID: \"06609e21-7e29-454a-a6ad-4dced184ed62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts" Nov 03 02:10:45.703689 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.703653 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncmtj\" (UniqueName: \"kubernetes.io/projected/06609e21-7e29-454a-a6ad-4dced184ed62-kube-api-access-ncmtj\") pod \"maas-api-revoke-sub-7eda9adc-cntts\" (UID: \"06609e21-7e29-454a-a6ad-4dced184ed62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts" Nov 03 02:10:45.759229 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.755051 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:45.825584 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:45.825494 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts" Nov 03 02:10:46.264665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:46.264633 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts"] Nov 03 02:10:46.265117 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:10:46.265089 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06609e21_7e29_454a_a6ad_4dced184ed62.slice/crio-191cc658fdfb6459a8e5c2e623b9da025ef10fbc5b43e453fbd83778e1bf0938 WatchSource:0}: Error finding container 191cc658fdfb6459a8e5c2e623b9da025ef10fbc5b43e453fbd83778e1bf0938: Status 404 returned error can't find the container with id 191cc658fdfb6459a8e5c2e623b9da025ef10fbc5b43e453fbd83778e1bf0938 Nov 03 02:10:46.309255 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:46.308259 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts" event={"ID":"06609e21-7e29-454a-a6ad-4dced184ed62","Type":"ContainerStarted","Data":"191cc658fdfb6459a8e5c2e623b9da025ef10fbc5b43e453fbd83778e1bf0938"} Nov 03 02:10:47.327484 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:47.323516 5463 generic.go:356] "Generic (PLEG): container finished" podID="06609e21-7e29-454a-a6ad-4dced184ed62" containerID="41a7835881912fc98ddffeed3855c4f8817c87fdc66a03266ab737c9a3d13943" exitCode=0 Nov 03 02:10:47.327484 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:47.323555 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts" event={"ID":"06609e21-7e29-454a-a6ad-4dced184ed62","Type":"ContainerDied","Data":"41a7835881912fc98ddffeed3855c4f8817c87fdc66a03266ab737c9a3d13943"} Nov 03 02:10:48.558740 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:48.558710 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts" Nov 03 02:10:48.650684 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:48.650647 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06609e21-7e29-454a-a6ad-4dced184ed62-kube-api-access-ncmtj\" (UniqueName: \"kubernetes.io/projected/06609e21-7e29-454a-a6ad-4dced184ed62-kube-api-access-ncmtj\") pod \"06609e21-7e29-454a-a6ad-4dced184ed62\" (UID: \"06609e21-7e29-454a-a6ad-4dced184ed62\") " Nov 03 02:10:48.650901 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:48.650847 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06609e21-7e29-454a-a6ad-4dced184ed62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06609e21-7e29-454a-a6ad-4dced184ed62-maas-api-openshift-ca\") pod \"06609e21-7e29-454a-a6ad-4dced184ed62\" (UID: \"06609e21-7e29-454a-a6ad-4dced184ed62\") " Nov 03 02:10:48.650968 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:48.650958 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06609e21-7e29-454a-a6ad-4dced184ed62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06609e21-7e29-454a-a6ad-4dced184ed62-maas-api-xks-ca\") pod \"06609e21-7e29-454a-a6ad-4dced184ed62\" (UID: \"06609e21-7e29-454a-a6ad-4dced184ed62\") " Nov 03 02:10:48.651240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:48.651175 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06609e21-7e29-454a-a6ad-4dced184ed62-maas-api-openshift-ca" pod "06609e21-7e29-454a-a6ad-4dced184ed62" (UID: "06609e21-7e29-454a-a6ad-4dced184ed62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:10:48.651484 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:48.651463 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06609e21-7e29-454a-a6ad-4dced184ed62-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:48.653140 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:48.653107 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06609e21-7e29-454a-a6ad-4dced184ed62-maas-api-xks-ca" pod "06609e21-7e29-454a-a6ad-4dced184ed62" (UID: "06609e21-7e29-454a-a6ad-4dced184ed62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:10:48.653435 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:48.653411 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06609e21-7e29-454a-a6ad-4dced184ed62-kube-api-access-ncmtj" pod "06609e21-7e29-454a-a6ad-4dced184ed62" (UID: "06609e21-7e29-454a-a6ad-4dced184ed62"). InnerVolumeSpecName "kube-api-access-ncmtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:48.753913 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:48.753842 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06609e21-7e29-454a-a6ad-4dced184ed62-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:48.753913 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:48.753882 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncmtj\" (UniqueName: \"kubernetes.io/projected/06609e21-7e29-454a-a6ad-4dced184ed62-kube-api-access-ncmtj\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:49.343135 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:49.343102 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts" Nov 03 02:10:49.343336 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:49.343130 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts" event={"ID":"06609e21-7e29-454a-a6ad-4dced184ed62","Type":"ContainerDied","Data":"191cc658fdfb6459a8e5c2e623b9da025ef10fbc5b43e453fbd83778e1bf0938"} Nov 03 02:10:49.343336 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:49.343185 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="191cc658fdfb6459a8e5c2e623b9da025ef10fbc5b43e453fbd83778e1bf0938" Nov 03 02:10:50.612629 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:50.612586 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:54.049113 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:54.049077 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84"] Nov 03 02:10:54.100808 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:54.100768 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c802d38-2cn84"] Nov 03 02:10:54.483175 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:54.482604 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4fd74ad-37f0-42d3-9d34-1f3779b90206" path="/var/lib/kubelet/pods/b4fd74ad-37f0-42d3-9d34-1f3779b90206/volumes" Nov 03 02:10:54.917214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:54.910173 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4"] Nov 03 02:10:54.935809 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:54.932368 5463 cpu_manager.go:414] "containerMap: removing container" podUID="06609e21-7e29-454a-a6ad-4dced184ed62" containerName="revoke-keys" Nov 03 02:10:54.935809 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:54.932395 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06609e21-7e29-454a-a6ad-4dced184ed62" containerName="revoke-keys" Nov 03 02:10:54.935809 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:54.932554 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="06609e21-7e29-454a-a6ad-4dced184ed62" containerName="revoke-keys" Nov 03 02:10:54.948364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:54.938716 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4" Nov 03 02:10:54.948364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:54.944752 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4"] Nov 03 02:10:55.012214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:55.009654 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:10:55.012214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:55.010441 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:10:55.012214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:55.010514 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:10:55.012214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:55.010580 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:10:55.124231 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:55.121646 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf59c59a-m8hd4\" (UID: \"b54649d6-2925-4feb-9f3f-ca3a91cec3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4" Nov 03 02:10:55.124231 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:55.121768 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf59c59a-m8hd4\" (UID: \"b54649d6-2925-4feb-9f3f-ca3a91cec3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4" Nov 03 02:10:55.124231 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:55.121908 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq5p8\" (UniqueName: \"kubernetes.io/projected/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-kube-api-access-pq5p8\") pod \"maas-api-revoke-sub-bf59c59a-m8hd4\" (UID: \"b54649d6-2925-4feb-9f3f-ca3a91cec3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4" Nov 03 02:10:55.289296 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:55.252147 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf59c59a-m8hd4\" (UID: \"b54649d6-2925-4feb-9f3f-ca3a91cec3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4" Nov 03 02:10:55.289296 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:55.252313 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq5p8\" (UniqueName: \"kubernetes.io/projected/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-kube-api-access-pq5p8\") pod \"maas-api-revoke-sub-bf59c59a-m8hd4\" (UID: \"b54649d6-2925-4feb-9f3f-ca3a91cec3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4" Nov 03 02:10:55.289296 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:55.252441 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf59c59a-m8hd4\" (UID: \"b54649d6-2925-4feb-9f3f-ca3a91cec3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4" Nov 03 02:10:55.289296 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:55.253387 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf59c59a-m8hd4\" (UID: \"b54649d6-2925-4feb-9f3f-ca3a91cec3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4" Nov 03 02:10:55.386227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:55.376676 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf59c59a-m8hd4\" (UID: \"b54649d6-2925-4feb-9f3f-ca3a91cec3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4" Nov 03 02:10:55.607673 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:55.607572 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:55.666024 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:55.665980 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq5p8\" (UniqueName: \"kubernetes.io/projected/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-kube-api-access-pq5p8\") pod \"maas-api-revoke-sub-bf59c59a-m8hd4\" (UID: \"b54649d6-2925-4feb-9f3f-ca3a91cec3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4" Nov 03 02:10:55.934312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:55.934277 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4" Nov 03 02:10:56.131324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:56.131245 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj"] Nov 03 02:10:56.194266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:56.193976 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecb34462-rp6gj"] Nov 03 02:10:56.194266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:56.194157 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:56.379356 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:56.379308 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 02:10:56.441330 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:56.441270 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:56.490261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:56.486335 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31542120-c0eb-48c7-9cae-7993a532e893" path="/var/lib/kubelet/pods/31542120-c0eb-48c7-9cae-7993a532e893/volumes" Nov 03 02:10:56.609121 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:56.583309 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 02:10:56.609121 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:56.583757 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:57.129270 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:57.082606 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:10:57.129270 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:57.082674 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" podUID="8d2c4397-7a8c-41a5-9140-2079f97bd460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:57.384265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:57.382133 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4"] Nov 03 02:10:57.431519 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:57.427733 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:57.457332 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:57.457300 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:10:57.483624 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:57.458669 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:57.609326 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:57.609295 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:10:57.628214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:57.596353 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:57.629715 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:57.600034 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:10:57.633292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:57.633254 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" podUID="32ef5394-19c0-4b33-84f6-2087dede03a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:57.637315 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:57.612565 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:10:57.637490 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:57.637464 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" podUID="731a025d-7302-4a22-8b43-70f96d8f05bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:57.641339 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:57.641314 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:10:57.658216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:57.596404 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:57.663337 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:57.663310 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:10:58.251452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:58.227290 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:58.410265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:58.379469 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:10:58.410265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:58.379531 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" podUID="1e5aa4d8-7e07-4efe-96ef-93b35748423e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:58.640292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:58.616764 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:58.712836 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:58.712793 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 02:10:58.890229 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:58.887453 5463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Nov 03 02:10:58.954572 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:58.910134 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:58.955124 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:58.910239 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:59.153243 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:59.137399 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:10:59.182884 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:59.182849 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4" event={"ID":"b54649d6-2925-4feb-9f3f-ca3a91cec3ff","Type":"ContainerStarted","Data":"1ed213f8161a5abbe9df36bfdc026f4d9a246b3d870fbae7f9282d568ef0c0bb"} Nov 03 02:10:59.263229 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:59.258714 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 03 02:10:59.263229 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:59.259147 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:10:59.349228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:59.345919 5463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:10:59.545342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:59.544750 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:10:59.652076 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:59.652046 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:10:59.655281 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:59.655260 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:10:59.732815 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:10:59.732789 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:11:00.192922 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.192832 5463 generic.go:356] "Generic (PLEG): container finished" podID="b54649d6-2925-4feb-9f3f-ca3a91cec3ff" containerID="3fb5a1323a2e66cb2a6a9c0e77584c18c7cdd3f4c73f0468344d656d7fbea5ad" exitCode=0 Nov 03 02:11:00.193093 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.192917 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4" event={"ID":"b54649d6-2925-4feb-9f3f-ca3a91cec3ff","Type":"ContainerDied","Data":"3fb5a1323a2e66cb2a6a9c0e77584c18c7cdd3f4c73f0468344d656d7fbea5ad"} Nov 03 02:11:00.544480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.544391 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7"] Nov 03 02:11:00.548704 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.548681 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.575042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.575017 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.575364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.575331 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.575364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.575363 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88714959-afe9-4294-b124-fca1f9d0c4f3-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.575552 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.575405 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.575552 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.575536 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.575665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.575608 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95vlf\" (UniqueName: \"kubernetes.io/projected/88714959-afe9-4294-b124-fca1f9d0c4f3-kube-api-access-95vlf\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.575731 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.575713 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.575796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.575745 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.575796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.575777 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.634835 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.634798 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7"] Nov 03 02:11:00.658812 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.658744 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:00.696600 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.696536 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.698547 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.698521 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95vlf\" (UniqueName: \"kubernetes.io/projected/88714959-afe9-4294-b124-fca1f9d0c4f3-kube-api-access-95vlf\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.698693 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.698632 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.698693 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.698657 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.698693 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.698682 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.698855 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.698807 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.698906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.698888 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.698954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.698916 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88714959-afe9-4294-b124-fca1f9d0c4f3-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.699009 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.698976 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.699492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.699466 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.700690 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.700081 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.701232 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.700920 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88714959-afe9-4294-b124-fca1f9d0c4f3-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.701232 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.701183 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.701833 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.701798 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.704590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.704546 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.712456 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.705158 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.761878 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.761849 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.775057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.775015 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95vlf\" (UniqueName: \"kubernetes.io/projected/88714959-afe9-4294-b124-fca1f9d0c4f3-kube-api-access-95vlf\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:00.866493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:00.866406 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:01.302526 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:11:01.302493 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88714959_afe9_4294_b124_fca1f9d0c4f3.slice/crio-3335eabe0ec96999418f0dca7ef5858a9e8f9dc2a999d73c1f4fb316a1f6466c WatchSource:0}: Error finding container 3335eabe0ec96999418f0dca7ef5858a9e8f9dc2a999d73c1f4fb316a1f6466c: Status 404 returned error can't find the container with id 3335eabe0ec96999418f0dca7ef5858a9e8f9dc2a999d73c1f4fb316a1f6466c Nov 03 02:11:01.305671 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:01.305265 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:01.305671 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:01.305342 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:01.305671 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:01.305373 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:01.307021 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:01.306999 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7"] Nov 03 02:11:01.387379 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:01.387341 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4" Nov 03 02:11:01.507622 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:01.507533 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-maas-api-openshift-ca\") pod \"b54649d6-2925-4feb-9f3f-ca3a91cec3ff\" (UID: \"b54649d6-2925-4feb-9f3f-ca3a91cec3ff\") " Nov 03 02:11:01.507779 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:01.507743 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-kube-api-access-pq5p8\" (UniqueName: \"kubernetes.io/projected/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-kube-api-access-pq5p8\") pod \"b54649d6-2925-4feb-9f3f-ca3a91cec3ff\" (UID: \"b54649d6-2925-4feb-9f3f-ca3a91cec3ff\") " Nov 03 02:11:01.507849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:01.507818 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-maas-api-xks-ca\") pod \"b54649d6-2925-4feb-9f3f-ca3a91cec3ff\" (UID: \"b54649d6-2925-4feb-9f3f-ca3a91cec3ff\") " Nov 03 02:11:01.508531 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:01.508503 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-maas-api-openshift-ca" pod "b54649d6-2925-4feb-9f3f-ca3a91cec3ff" (UID: "b54649d6-2925-4feb-9f3f-ca3a91cec3ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:11:01.509007 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:01.508985 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:01.510926 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:01.510830 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-kube-api-access-pq5p8" pod "b54649d6-2925-4feb-9f3f-ca3a91cec3ff" (UID: "b54649d6-2925-4feb-9f3f-ca3a91cec3ff"). InnerVolumeSpecName "kube-api-access-pq5p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:11:01.510926 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:01.510890 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-maas-api-xks-ca" pod "b54649d6-2925-4feb-9f3f-ca3a91cec3ff" (UID: "b54649d6-2925-4feb-9f3f-ca3a91cec3ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:11:01.610499 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:01.610465 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq5p8\" (UniqueName: \"kubernetes.io/projected/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-kube-api-access-pq5p8\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:01.610499 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:01.610495 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b54649d6-2925-4feb-9f3f-ca3a91cec3ff-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:02.211703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:02.211672 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4" Nov 03 02:11:02.211878 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:02.211698 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4" event={"ID":"b54649d6-2925-4feb-9f3f-ca3a91cec3ff","Type":"ContainerDied","Data":"1ed213f8161a5abbe9df36bfdc026f4d9a246b3d870fbae7f9282d568ef0c0bb"} Nov 03 02:11:02.211878 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:02.211757 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ed213f8161a5abbe9df36bfdc026f4d9a246b3d870fbae7f9282d568ef0c0bb" Nov 03 02:11:02.213571 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:02.213546 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" event={"ID":"88714959-afe9-4294-b124-fca1f9d0c4f3","Type":"ContainerStarted","Data":"f7907928918c239aa4669ee8f9aa050a50436457c7e0181fa72319d611a6eae5"} Nov 03 02:11:02.213691 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:02.213587 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" event={"ID":"88714959-afe9-4294-b124-fca1f9d0c4f3","Type":"ContainerStarted","Data":"3335eabe0ec96999418f0dca7ef5858a9e8f9dc2a999d73c1f4fb316a1f6466c"} Nov 03 02:11:02.387312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:02.387253 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" podStartSLOduration=2.387236635 podStartE2EDuration="2.387236635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:11:02.382738719 +0000 UTC m=+1144.066847875" watchObservedRunningTime="2026-11-03 02:11:02.387236635 +0000 UTC m=+1144.071345790" Nov 03 02:11:02.870386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:02.869874 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:02.870386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:02.870034 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 03 02:11:02.870386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:02.870076 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" podUID="88714959-afe9-4294-b124-fca1f9d0c4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 03 02:11:03.572570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.572535 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4"] Nov 03 02:11:03.573361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.573338 5463 cpu_manager.go:414] "containerMap: removing container" podUID="b54649d6-2925-4feb-9f3f-ca3a91cec3ff" containerName="revoke-keys" Nov 03 02:11:03.573361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.573358 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b54649d6-2925-4feb-9f3f-ca3a91cec3ff" containerName="revoke-keys" Nov 03 02:11:03.573552 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.573482 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b54649d6-2925-4feb-9f3f-ca3a91cec3ff" containerName="revoke-keys" Nov 03 02:11:03.577604 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.577585 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4" Nov 03 02:11:03.611773 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.611744 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4"] Nov 03 02:11:03.634091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.634061 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b17eb9a-2379-441c-8d88-3b6d1148462c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd336ebf-rdzq4\" (UID: \"2b17eb9a-2379-441c-8d88-3b6d1148462c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4" Nov 03 02:11:03.634446 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.634393 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b17eb9a-2379-441c-8d88-3b6d1148462c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd336ebf-rdzq4\" (UID: \"2b17eb9a-2379-441c-8d88-3b6d1148462c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4" Nov 03 02:11:03.634620 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.634605 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pgfd\" (UniqueName: \"kubernetes.io/projected/2b17eb9a-2379-441c-8d88-3b6d1148462c-kube-api-access-4pgfd\") pod \"maas-api-revoke-sub-dd336ebf-rdzq4\" (UID: \"2b17eb9a-2379-441c-8d88-3b6d1148462c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4" Nov 03 02:11:03.672554 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.672525 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:11:03.672855 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.672838 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:11:03.672992 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.672979 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:11:03.683216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.680693 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:11:03.736467 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.736444 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b17eb9a-2379-441c-8d88-3b6d1148462c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd336ebf-rdzq4\" (UID: \"2b17eb9a-2379-441c-8d88-3b6d1148462c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4" Nov 03 02:11:03.736571 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.736502 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b17eb9a-2379-441c-8d88-3b6d1148462c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd336ebf-rdzq4\" (UID: \"2b17eb9a-2379-441c-8d88-3b6d1148462c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4" Nov 03 02:11:03.736571 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.736542 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pgfd\" (UniqueName: \"kubernetes.io/projected/2b17eb9a-2379-441c-8d88-3b6d1148462c-kube-api-access-4pgfd\") pod \"maas-api-revoke-sub-dd336ebf-rdzq4\" (UID: \"2b17eb9a-2379-441c-8d88-3b6d1148462c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4" Nov 03 02:11:03.737277 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.737255 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b17eb9a-2379-441c-8d88-3b6d1148462c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd336ebf-rdzq4\" (UID: \"2b17eb9a-2379-441c-8d88-3b6d1148462c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4" Nov 03 02:11:03.738720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.738698 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b17eb9a-2379-441c-8d88-3b6d1148462c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd336ebf-rdzq4\" (UID: \"2b17eb9a-2379-441c-8d88-3b6d1148462c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4" Nov 03 02:11:03.793823 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.793798 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pgfd\" (UniqueName: \"kubernetes.io/projected/2b17eb9a-2379-441c-8d88-3b6d1148462c-kube-api-access-4pgfd\") pod \"maas-api-revoke-sub-dd336ebf-rdzq4\" (UID: \"2b17eb9a-2379-441c-8d88-3b6d1148462c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4" Nov 03 02:11:03.872666 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.872574 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 03 02:11:03.872666 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.872629 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" podUID="88714959-afe9-4294-b124-fca1f9d0c4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 03 02:11:03.891148 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:03.891125 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4" Nov 03 02:11:04.284609 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:11:04.284576 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b17eb9a_2379_441c_8d88_3b6d1148462c.slice/crio-ed97590f1f3f6704b354d10167ce6adecd1e9d033330d7463190c3434f721bd9 WatchSource:0}: Error finding container ed97590f1f3f6704b354d10167ce6adecd1e9d033330d7463190c3434f721bd9: Status 404 returned error can't find the container with id ed97590f1f3f6704b354d10167ce6adecd1e9d033330d7463190c3434f721bd9 Nov 03 02:11:04.289382 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:04.289357 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4"] Nov 03 02:11:04.874240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:04.869022 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 03 02:11:04.874240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:04.869080 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" podUID="88714959-afe9-4294-b124-fca1f9d0c4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 03 02:11:05.235375 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:05.235340 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4" event={"ID":"2b17eb9a-2379-441c-8d88-3b6d1148462c","Type":"ContainerStarted","Data":"5ddf65b000bd2ba7004d83f0a4f3c08f651fb17a6b6ca68f2f9fb10f57c7b3af"} Nov 03 02:11:05.235580 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:05.235398 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4" event={"ID":"2b17eb9a-2379-441c-8d88-3b6d1148462c","Type":"ContainerStarted","Data":"ed97590f1f3f6704b354d10167ce6adecd1e9d033330d7463190c3434f721bd9"} Nov 03 02:11:05.374249 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:05.367108 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4" podStartSLOduration=2.367090222 podStartE2EDuration="2.367090222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:11:05.351570658 +0000 UTC m=+1147.035679818" watchObservedRunningTime="2026-11-03 02:11:05.367090222 +0000 UTC m=+1147.051199378" Nov 03 02:11:05.477894 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:05.477845 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:05.868315 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:05.868276 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 03 02:11:05.868556 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:05.868528 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" podUID="88714959-afe9-4294-b124-fca1f9d0c4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 03 02:11:06.244827 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:06.244790 5463 generic.go:356] "Generic (PLEG): container finished" podID="2b17eb9a-2379-441c-8d88-3b6d1148462c" containerID="5ddf65b000bd2ba7004d83f0a4f3c08f651fb17a6b6ca68f2f9fb10f57c7b3af" exitCode=0 Nov 03 02:11:06.245264 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:06.244878 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4" event={"ID":"2b17eb9a-2379-441c-8d88-3b6d1148462c","Type":"ContainerDied","Data":"5ddf65b000bd2ba7004d83f0a4f3c08f651fb17a6b6ca68f2f9fb10f57c7b3af"} Nov 03 02:11:06.868526 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:06.868488 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 03 02:11:06.868811 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:06.868783 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" podUID="88714959-afe9-4294-b124-fca1f9d0c4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 03 02:11:07.470462 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:07.470435 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4" Nov 03 02:11:07.613119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:07.613019 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b17eb9a-2379-441c-8d88-3b6d1148462c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b17eb9a-2379-441c-8d88-3b6d1148462c-maas-api-xks-ca\") pod \"2b17eb9a-2379-441c-8d88-3b6d1148462c\" (UID: \"2b17eb9a-2379-441c-8d88-3b6d1148462c\") " Nov 03 02:11:07.613119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:07.613067 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b17eb9a-2379-441c-8d88-3b6d1148462c-kube-api-access-4pgfd\" (UniqueName: \"kubernetes.io/projected/2b17eb9a-2379-441c-8d88-3b6d1148462c-kube-api-access-4pgfd\") pod \"2b17eb9a-2379-441c-8d88-3b6d1148462c\" (UID: \"2b17eb9a-2379-441c-8d88-3b6d1148462c\") " Nov 03 02:11:07.613386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:07.613132 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b17eb9a-2379-441c-8d88-3b6d1148462c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b17eb9a-2379-441c-8d88-3b6d1148462c-maas-api-openshift-ca\") pod \"2b17eb9a-2379-441c-8d88-3b6d1148462c\" (UID: \"2b17eb9a-2379-441c-8d88-3b6d1148462c\") " Nov 03 02:11:07.613593 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:07.613564 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b17eb9a-2379-441c-8d88-3b6d1148462c-maas-api-openshift-ca" pod "2b17eb9a-2379-441c-8d88-3b6d1148462c" (UID: "2b17eb9a-2379-441c-8d88-3b6d1148462c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:11:07.613848 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:07.613824 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b17eb9a-2379-441c-8d88-3b6d1148462c-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:07.615938 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:07.615907 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b17eb9a-2379-441c-8d88-3b6d1148462c-kube-api-access-4pgfd" pod "2b17eb9a-2379-441c-8d88-3b6d1148462c" (UID: "2b17eb9a-2379-441c-8d88-3b6d1148462c"). InnerVolumeSpecName "kube-api-access-4pgfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:11:07.616167 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:07.616143 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b17eb9a-2379-441c-8d88-3b6d1148462c-maas-api-xks-ca" pod "2b17eb9a-2379-441c-8d88-3b6d1148462c" (UID: "2b17eb9a-2379-441c-8d88-3b6d1148462c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:11:07.715857 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:07.715821 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b17eb9a-2379-441c-8d88-3b6d1148462c-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:07.715857 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:07.715862 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pgfd\" (UniqueName: \"kubernetes.io/projected/2b17eb9a-2379-441c-8d88-3b6d1148462c-kube-api-access-4pgfd\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:07.867472 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:07.867390 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 03 02:11:07.867472 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:07.867456 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" podUID="88714959-afe9-4294-b124-fca1f9d0c4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 03 02:11:08.274207 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:08.274165 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4" Nov 03 02:11:08.274383 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:08.274215 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4" event={"ID":"2b17eb9a-2379-441c-8d88-3b6d1148462c","Type":"ContainerDied","Data":"ed97590f1f3f6704b354d10167ce6adecd1e9d033330d7463190c3434f721bd9"} Nov 03 02:11:08.274383 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:08.274273 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed97590f1f3f6704b354d10167ce6adecd1e9d033330d7463190c3434f721bd9" Nov 03 02:11:08.872101 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:08.872074 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:08.872535 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:08.872516 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:08.873253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:08.873226 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:11:09.763041 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:09.763008 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:11.186338 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:11.186305 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:11.350894 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:11.350866 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:11.508382 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:11.508297 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:12.883093 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:12.883048 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv"] Nov 03 02:11:12.883712 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:12.883695 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2b17eb9a-2379-441c-8d88-3b6d1148462c" containerName="revoke-keys" Nov 03 02:11:12.883712 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:12.883712 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b17eb9a-2379-441c-8d88-3b6d1148462c" containerName="revoke-keys" Nov 03 02:11:12.883852 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:12.883840 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b17eb9a-2379-441c-8d88-3b6d1148462c" containerName="revoke-keys" Nov 03 02:11:12.887718 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:12.887694 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:12.980923 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:12.980880 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:12.980923 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:12.980933 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:12.981216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:12.981003 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:12.981216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:12.981064 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:12.981216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:12.981090 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:12.981376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:12.981303 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z9t9\" (UniqueName: \"kubernetes.io/projected/30a0bcec-82be-4efd-bd03-87dadb66b21b-kube-api-access-5z9t9\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:12.981599 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:12.981569 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:12.981668 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:12.981613 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30a0bcec-82be-4efd-bd03-87dadb66b21b-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:12.981733 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:12.981687 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:13.012398 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:11:13.012350 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:11:13.083721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.082766 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z9t9\" (UniqueName: \"kubernetes.io/projected/30a0bcec-82be-4efd-bd03-87dadb66b21b-kube-api-access-5z9t9\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:13.083721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.082852 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:13.083721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.082876 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30a0bcec-82be-4efd-bd03-87dadb66b21b-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:13.083721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.082914 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:13.083721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.083039 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:13.083721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.083061 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:13.083721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.083122 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:13.083721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.083181 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:13.083721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.083220 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:13.087221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.085601 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:13.087221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.085758 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:13.087221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.085899 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:13.087221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.086039 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:13.087221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.086525 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30a0bcec-82be-4efd-bd03-87dadb66b21b-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:13.087927 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.087742 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:13.091987 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.091951 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:13.093407 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.093387 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv"] Nov 03 02:11:13.141456 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.141391 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:13.141587 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.141472 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z9t9\" (UniqueName: \"kubernetes.io/projected/30a0bcec-82be-4efd-bd03-87dadb66b21b-kube-api-access-5z9t9\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:13.205116 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.204823 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:13.560434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.560402 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6"] Nov 03 02:11:13.565832 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.565813 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6" Nov 03 02:11:13.616418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.616393 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6"] Nov 03 02:11:13.618783 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.618755 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:11:13.619042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.619016 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:11:13.619147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.619047 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:11:13.619147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.619064 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:11:13.626418 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:11:13.626390 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a0bcec_82be_4efd_bd03_87dadb66b21b.slice/crio-a9788f5fdfda26bf8a6f3a69ca111e116bad9f9598e34cf46c317498b753f432 WatchSource:0}: Error finding container a9788f5fdfda26bf8a6f3a69ca111e116bad9f9598e34cf46c317498b753f432: Status 404 returned error can't find the container with id a9788f5fdfda26bf8a6f3a69ca111e116bad9f9598e34cf46c317498b753f432 Nov 03 02:11:13.629505 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.629471 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:13.629589 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.629556 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:13.629638 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.629605 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:13.632644 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.632622 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv"] Nov 03 02:11:13.691772 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.691745 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l49z\" (UniqueName: \"kubernetes.io/projected/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-kube-api-access-2l49z\") pod \"maas-api-revoke-sub-b4fa141e-6h4p6\" (UID: \"1d7258ba-d952-4fd7-8e9f-9d1cae62de09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6" Nov 03 02:11:13.691894 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.691875 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4fa141e-6h4p6\" (UID: \"1d7258ba-d952-4fd7-8e9f-9d1cae62de09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6" Nov 03 02:11:13.692149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.692128 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4fa141e-6h4p6\" (UID: \"1d7258ba-d952-4fd7-8e9f-9d1cae62de09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6" Nov 03 02:11:13.793253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.793218 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4fa141e-6h4p6\" (UID: \"1d7258ba-d952-4fd7-8e9f-9d1cae62de09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6" Nov 03 02:11:13.793411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.793281 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l49z\" (UniqueName: \"kubernetes.io/projected/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-kube-api-access-2l49z\") pod \"maas-api-revoke-sub-b4fa141e-6h4p6\" (UID: \"1d7258ba-d952-4fd7-8e9f-9d1cae62de09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6" Nov 03 02:11:13.793411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.793338 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4fa141e-6h4p6\" (UID: \"1d7258ba-d952-4fd7-8e9f-9d1cae62de09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6" Nov 03 02:11:13.794110 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.794083 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4fa141e-6h4p6\" (UID: \"1d7258ba-d952-4fd7-8e9f-9d1cae62de09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6" Nov 03 02:11:13.796007 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.795979 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4fa141e-6h4p6\" (UID: \"1d7258ba-d952-4fd7-8e9f-9d1cae62de09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6" Nov 03 02:11:13.850527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.850457 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l49z\" (UniqueName: \"kubernetes.io/projected/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-kube-api-access-2l49z\") pod \"maas-api-revoke-sub-b4fa141e-6h4p6\" (UID: \"1d7258ba-d952-4fd7-8e9f-9d1cae62de09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6" Nov 03 02:11:13.880278 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:13.880251 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6" Nov 03 02:11:14.275932 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:11:14.275889 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7258ba_d952_4fd7_8e9f_9d1cae62de09.slice/crio-286c697cfdd7327992bf7ed7902508089d80d8d232d5e3fc9501a3c7b936a673 WatchSource:0}: Error finding container 286c697cfdd7327992bf7ed7902508089d80d8d232d5e3fc9501a3c7b936a673: Status 404 returned error can't find the container with id 286c697cfdd7327992bf7ed7902508089d80d8d232d5e3fc9501a3c7b936a673 Nov 03 02:11:14.280356 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:14.280332 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6"] Nov 03 02:11:14.321906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:14.321880 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" event={"ID":"30a0bcec-82be-4efd-bd03-87dadb66b21b","Type":"ContainerStarted","Data":"4ea19b22cb34eb9d62384865369b7a81ebabed16b32a994dde67b2e4ca656637"} Nov 03 02:11:14.322015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:14.321930 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" event={"ID":"30a0bcec-82be-4efd-bd03-87dadb66b21b","Type":"ContainerStarted","Data":"a9788f5fdfda26bf8a6f3a69ca111e116bad9f9598e34cf46c317498b753f432"} Nov 03 02:11:14.323384 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:14.323363 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6" event={"ID":"1d7258ba-d952-4fd7-8e9f-9d1cae62de09","Type":"ContainerStarted","Data":"286c697cfdd7327992bf7ed7902508089d80d8d232d5e3fc9501a3c7b936a673"} Nov 03 02:11:14.444174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:14.444123 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" podStartSLOduration=2.444108424 podStartE2EDuration="2.444108424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:11:14.439306171 +0000 UTC m=+1156.123415325" watchObservedRunningTime="2026-11-03 02:11:14.444108424 +0000 UTC m=+1156.128217579" Nov 03 02:11:15.205906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.205870 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:15.206816 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.206796 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 02:11:15.206901 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.206844 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" podUID="30a0bcec-82be-4efd-bd03-87dadb66b21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 02:11:15.242834 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.242809 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:15.343954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.343917 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6" event={"ID":"1d7258ba-d952-4fd7-8e9f-9d1cae62de09","Type":"ContainerStarted","Data":"4e3f3e8d6220b64067c73f3e35dfa8e8ea9a4e14bd1da21d85dae7379aa7f717"} Nov 03 02:11:15.481430 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.481328 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6" podStartSLOduration=2.48131287 podStartE2EDuration="2.48131287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:11:15.469461773 +0000 UTC m=+1157.153570928" watchObservedRunningTime="2026-11-03 02:11:15.48131287 +0000 UTC m=+1157.165422026" Nov 03 02:11:15.683218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.665051 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx"] Nov 03 02:11:15.687557 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.687519 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.745902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.739749 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx"] Nov 03 02:11:15.823469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.823427 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.823658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.823487 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j55js\" (UniqueName: \"kubernetes.io/projected/2617013b-bed0-446d-a597-31bb323d1c6a-kube-api-access-j55js\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.823658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.823518 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.823658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.823546 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.823658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.823598 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.823848 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.823708 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2617013b-bed0-446d-a597-31bb323d1c6a-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.823848 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.823743 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2617013b-bed0-446d-a597-31bb323d1c6a-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.823848 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.823780 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.825911 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.825859 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2617013b-bed0-446d-a597-31bb323d1c6a-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.927007 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.926968 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.927222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.927091 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2617013b-bed0-446d-a597-31bb323d1c6a-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.927222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.927128 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2617013b-bed0-446d-a597-31bb323d1c6a-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.927222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.927161 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.927400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.927268 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2617013b-bed0-446d-a597-31bb323d1c6a-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.927461 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.927398 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.927461 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.927437 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j55js\" (UniqueName: \"kubernetes.io/projected/2617013b-bed0-446d-a597-31bb323d1c6a-kube-api-access-j55js\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.927565 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.927467 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.927565 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.927494 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.927665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.927602 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.927804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.927778 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.927874 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.927852 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.927965 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.927941 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.928176 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.928159 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2617013b-bed0-446d-a597-31bb323d1c6a-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.929291 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.929264 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.929729 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.929706 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2617013b-bed0-446d-a597-31bb323d1c6a-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.989213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.988995 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2617013b-bed0-446d-a597-31bb323d1c6a-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:15.992874 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:15.991425 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j55js\" (UniqueName: \"kubernetes.io/projected/2617013b-bed0-446d-a597-31bb323d1c6a-kube-api-access-j55js\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:16.013415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:16.013358 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:16.205248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:16.205214 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 02:11:16.205409 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:16.205263 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" podUID="30a0bcec-82be-4efd-bd03-87dadb66b21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 02:11:16.380747 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:16.380663 5463 generic.go:356] "Generic (PLEG): container finished" podID="1d7258ba-d952-4fd7-8e9f-9d1cae62de09" containerID="4e3f3e8d6220b64067c73f3e35dfa8e8ea9a4e14bd1da21d85dae7379aa7f717" exitCode=0 Nov 03 02:11:16.381583 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:16.381555 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6" event={"ID":"1d7258ba-d952-4fd7-8e9f-9d1cae62de09","Type":"ContainerDied","Data":"4e3f3e8d6220b64067c73f3e35dfa8e8ea9a4e14bd1da21d85dae7379aa7f717"} Nov 03 02:11:16.481524 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:16.481493 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx"] Nov 03 02:11:16.484295 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:16.484253 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:16.484396 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:16.484335 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:16.484396 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:16.484382 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:11:17.038570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:17.038530 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:17.180765 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:17.180722 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:17.205325 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:17.205295 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 02:11:17.205455 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:17.205349 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" podUID="30a0bcec-82be-4efd-bd03-87dadb66b21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 02:11:17.422592 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:17.422558 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" event={"ID":"2617013b-bed0-446d-a597-31bb323d1c6a","Type":"ContainerStarted","Data":"d47147148bca0b49acfcb877762bf35db07ee00784f8854099753e9fc5625abf"} Nov 03 02:11:17.423035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:17.423017 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" event={"ID":"2617013b-bed0-446d-a597-31bb323d1c6a","Type":"ContainerStarted","Data":"e4e4faffbe789e4b8104a7da712f774ebf3db094ddbc71a1c5c679350154dbe2"} Nov 03 02:11:17.546232 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:17.544060 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" podStartSLOduration=2.544046613 podStartE2EDuration="2.544046613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:11:17.541073155 +0000 UTC m=+1159.225182307" watchObservedRunningTime="2026-11-03 02:11:17.544046613 +0000 UTC m=+1159.228155768" Nov 03 02:11:17.840143 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:17.840112 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6" Nov 03 02:11:17.842268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:17.842238 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-kube-api-access-2l49z\" (UniqueName: \"kubernetes.io/projected/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-kube-api-access-2l49z\") pod \"1d7258ba-d952-4fd7-8e9f-9d1cae62de09\" (UID: \"1d7258ba-d952-4fd7-8e9f-9d1cae62de09\") " Nov 03 02:11:17.842406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:17.842368 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-maas-api-xks-ca\") pod \"1d7258ba-d952-4fd7-8e9f-9d1cae62de09\" (UID: \"1d7258ba-d952-4fd7-8e9f-9d1cae62de09\") " Nov 03 02:11:17.842591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:17.842574 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-maas-api-openshift-ca\") pod \"1d7258ba-d952-4fd7-8e9f-9d1cae62de09\" (UID: \"1d7258ba-d952-4fd7-8e9f-9d1cae62de09\") " Nov 03 02:11:17.843591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:17.843566 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-maas-api-openshift-ca" pod "1d7258ba-d952-4fd7-8e9f-9d1cae62de09" (UID: "1d7258ba-d952-4fd7-8e9f-9d1cae62de09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:11:17.845831 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:17.845803 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-kube-api-access-2l49z" pod "1d7258ba-d952-4fd7-8e9f-9d1cae62de09" (UID: "1d7258ba-d952-4fd7-8e9f-9d1cae62de09"). InnerVolumeSpecName "kube-api-access-2l49z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:11:17.846182 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:17.846157 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-maas-api-xks-ca" pod "1d7258ba-d952-4fd7-8e9f-9d1cae62de09" (UID: "1d7258ba-d952-4fd7-8e9f-9d1cae62de09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:11:17.944028 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:17.943996 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:17.944028 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:17.944021 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l49z\" (UniqueName: \"kubernetes.io/projected/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-kube-api-access-2l49z\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:17.944028 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:17.944032 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7258ba-d952-4fd7-8e9f-9d1cae62de09-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:18.017818 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.017730 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:18.019523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.019493 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 02:11:18.019721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.019695 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" podUID="2617013b-bed0-446d-a597-31bb323d1c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 02:11:18.209227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.208611 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 02:11:18.209227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.208675 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" podUID="30a0bcec-82be-4efd-bd03-87dadb66b21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 02:11:18.408930 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.408890 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h"] Nov 03 02:11:18.411238 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.411215 5463 cpu_manager.go:414] "containerMap: removing container" podUID="1d7258ba-d952-4fd7-8e9f-9d1cae62de09" containerName="revoke-keys" Nov 03 02:11:18.411392 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.411378 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7258ba-d952-4fd7-8e9f-9d1cae62de09" containerName="revoke-keys" Nov 03 02:11:18.411665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.411652 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d7258ba-d952-4fd7-8e9f-9d1cae62de09" containerName="revoke-keys" Nov 03 02:11:18.415869 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.415851 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h" Nov 03 02:11:18.434468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.434444 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6" Nov 03 02:11:18.434468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.434456 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6" event={"ID":"1d7258ba-d952-4fd7-8e9f-9d1cae62de09","Type":"ContainerDied","Data":"286c697cfdd7327992bf7ed7902508089d80d8d232d5e3fc9501a3c7b936a673"} Nov 03 02:11:18.434849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.434499 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="286c697cfdd7327992bf7ed7902508089d80d8d232d5e3fc9501a3c7b936a673" Nov 03 02:11:18.494739 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.492393 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28dab9fe-c0a9-465c-98ac-759f60d92cbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aa43ebe-rmm7h\" (UID: \"28dab9fe-c0a9-465c-98ac-759f60d92cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h" Nov 03 02:11:18.494739 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.492566 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt6l5\" (UniqueName: \"kubernetes.io/projected/28dab9fe-c0a9-465c-98ac-759f60d92cbf-kube-api-access-vt6l5\") pod \"maas-api-revoke-sub-9aa43ebe-rmm7h\" (UID: \"28dab9fe-c0a9-465c-98ac-759f60d92cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h" Nov 03 02:11:18.494739 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.492631 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28dab9fe-c0a9-465c-98ac-759f60d92cbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aa43ebe-rmm7h\" (UID: \"28dab9fe-c0a9-465c-98ac-759f60d92cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h" Nov 03 02:11:18.525211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.513028 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h"] Nov 03 02:11:18.618236 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.611527 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28dab9fe-c0a9-465c-98ac-759f60d92cbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aa43ebe-rmm7h\" (UID: \"28dab9fe-c0a9-465c-98ac-759f60d92cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h" Nov 03 02:11:18.618236 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.611645 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt6l5\" (UniqueName: \"kubernetes.io/projected/28dab9fe-c0a9-465c-98ac-759f60d92cbf-kube-api-access-vt6l5\") pod \"maas-api-revoke-sub-9aa43ebe-rmm7h\" (UID: \"28dab9fe-c0a9-465c-98ac-759f60d92cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h" Nov 03 02:11:18.618236 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.611715 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28dab9fe-c0a9-465c-98ac-759f60d92cbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aa43ebe-rmm7h\" (UID: \"28dab9fe-c0a9-465c-98ac-759f60d92cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h" Nov 03 02:11:18.618236 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.612800 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28dab9fe-c0a9-465c-98ac-759f60d92cbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aa43ebe-rmm7h\" (UID: \"28dab9fe-c0a9-465c-98ac-759f60d92cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h" Nov 03 02:11:18.625678 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.625640 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28dab9fe-c0a9-465c-98ac-759f60d92cbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aa43ebe-rmm7h\" (UID: \"28dab9fe-c0a9-465c-98ac-759f60d92cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h" Nov 03 02:11:18.734023 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:18.733924 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt6l5\" (UniqueName: \"kubernetes.io/projected/28dab9fe-c0a9-465c-98ac-759f60d92cbf-kube-api-access-vt6l5\") pod \"maas-api-revoke-sub-9aa43ebe-rmm7h\" (UID: \"28dab9fe-c0a9-465c-98ac-759f60d92cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h" Nov 03 02:11:19.014265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:19.013984 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 02:11:19.014265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:19.014053 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" podUID="2617013b-bed0-446d-a597-31bb323d1c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 02:11:19.042783 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:19.042747 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h" Nov 03 02:11:19.207489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:19.205508 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 02:11:19.207489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:19.205567 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" podUID="30a0bcec-82be-4efd-bd03-87dadb66b21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 02:11:19.617043 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:19.617007 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h"] Nov 03 02:11:19.617503 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:11:19.617416 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28dab9fe_c0a9_465c_98ac_759f60d92cbf.slice/crio-814d79bd0bab23e67ef7dd2ce6b81a8bbaa82384cbbd79acc98553d985f0d544 WatchSource:0}: Error finding container 814d79bd0bab23e67ef7dd2ce6b81a8bbaa82384cbbd79acc98553d985f0d544: Status 404 returned error can't find the container with id 814d79bd0bab23e67ef7dd2ce6b81a8bbaa82384cbbd79acc98553d985f0d544 Nov 03 02:11:20.014684 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:20.014313 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 02:11:20.014684 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:20.014369 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" podUID="2617013b-bed0-446d-a597-31bb323d1c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 02:11:20.205465 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:20.205426 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 02:11:20.205667 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:20.205487 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" podUID="30a0bcec-82be-4efd-bd03-87dadb66b21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 02:11:20.208350 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:20.208291 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:20.546823 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:20.530384 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h" event={"ID":"28dab9fe-c0a9-465c-98ac-759f60d92cbf","Type":"ContainerStarted","Data":"2827fdf0c5c39719805bd689e30d7a12423a75bcb874f5d918a08ea5b18af63e"} Nov 03 02:11:20.546823 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:20.530458 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h" event={"ID":"28dab9fe-c0a9-465c-98ac-759f60d92cbf","Type":"ContainerStarted","Data":"814d79bd0bab23e67ef7dd2ce6b81a8bbaa82384cbbd79acc98553d985f0d544"} Nov 03 02:11:20.687215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:20.686398 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h" podStartSLOduration=2.686375199 podStartE2EDuration="2.686375199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:11:20.647698357 +0000 UTC m=+1162.331807514" watchObservedRunningTime="2026-11-03 02:11:20.686375199 +0000 UTC m=+1162.370484355" Nov 03 02:11:21.018210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:21.017596 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 02:11:21.018210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:21.017657 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" podUID="2617013b-bed0-446d-a597-31bb323d1c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 02:11:21.648224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:21.647682 5463 generic.go:356] "Generic (PLEG): container finished" podID="28dab9fe-c0a9-465c-98ac-759f60d92cbf" containerID="2827fdf0c5c39719805bd689e30d7a12423a75bcb874f5d918a08ea5b18af63e" exitCode=0 Nov 03 02:11:21.648224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:21.647838 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h" event={"ID":"28dab9fe-c0a9-465c-98ac-759f60d92cbf","Type":"ContainerDied","Data":"2827fdf0c5c39719805bd689e30d7a12423a75bcb874f5d918a08ea5b18af63e"} Nov 03 02:11:21.686214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:21.667315 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:21.741215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:21.737703 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 03 02:11:21.961640 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:21.961538 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 02:11:22.040218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:22.039426 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 02:11:22.040218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:22.039481 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" podUID="2617013b-bed0-446d-a597-31bb323d1c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 02:11:22.140258 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:22.139786 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:22.250375 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:22.249928 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:11:22.250375 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:22.249987 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" podUID="30a0bcec-82be-4efd-bd03-87dadb66b21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:22.670218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:22.667724 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 03 02:11:22.670218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:22.669900 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 02:11:22.678213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:22.674277 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:11:22.980270 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:22.971003 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:22.980270 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:22.971445 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:11:23.057311 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:23.057256 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 02:11:23.082641 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:23.072947 5463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:23.083013 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:23.082477 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" podUID="2617013b-bed0-446d-a597-31bb323d1c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 02:11:23.170228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:23.147330 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 03 02:11:23.198216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:23.172648 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:11:23.302124 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:23.302032 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:11:23.419218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:23.346159 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" podUID="30a0bcec-82be-4efd-bd03-87dadb66b21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:23.713672 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:23.713640 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:23.744372 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:23.744338 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:23.800217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:23.764182 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 03 02:11:24.184561 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:23.803018 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:11:24.406215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:23.904355 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 02:11:24.406215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:23.943316 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:24.406215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:24.404827 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:11:24.406215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:23.984278 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:24.406587 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:24.016510 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:11:24.406650 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:24.406591 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" podUID="dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:24.406650 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:24.087344 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 02:11:24.406766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:24.406661 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" podUID="2617013b-bed0-446d-a597-31bb323d1c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 02:11:24.406766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:24.175497 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:25.037236 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:25.027115 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 02:11:25.037236 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:25.027204 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" podUID="2617013b-bed0-446d-a597-31bb323d1c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 02:11:25.443876 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:25.443817 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 02:11:25.444797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:25.444540 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 02:11:25.448213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:25.444577 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 03 02:11:25.448431 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:25.444619 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 03 02:11:25.562217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:25.558414 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:11:25.562217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:25.558566 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" podUID="88714959-afe9-4294-b124-fca1f9d0c4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:25.604366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:25.604313 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:25.742276 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:25.741369 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h" Nov 03 02:11:25.841592 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:25.841558 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28dab9fe-c0a9-465c-98ac-759f60d92cbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28dab9fe-c0a9-465c-98ac-759f60d92cbf-maas-api-xks-ca\") pod \"28dab9fe-c0a9-465c-98ac-759f60d92cbf\" (UID: \"28dab9fe-c0a9-465c-98ac-759f60d92cbf\") " Nov 03 02:11:25.841935 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:25.841910 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28dab9fe-c0a9-465c-98ac-759f60d92cbf-kube-api-access-vt6l5\" (UniqueName: \"kubernetes.io/projected/28dab9fe-c0a9-465c-98ac-759f60d92cbf-kube-api-access-vt6l5\") pod \"28dab9fe-c0a9-465c-98ac-759f60d92cbf\" (UID: \"28dab9fe-c0a9-465c-98ac-759f60d92cbf\") " Nov 03 02:11:25.845404 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:25.845376 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28dab9fe-c0a9-465c-98ac-759f60d92cbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28dab9fe-c0a9-465c-98ac-759f60d92cbf-maas-api-openshift-ca\") pod \"28dab9fe-c0a9-465c-98ac-759f60d92cbf\" (UID: \"28dab9fe-c0a9-465c-98ac-759f60d92cbf\") " Nov 03 02:11:25.997448 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:25.888497 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28dab9fe-c0a9-465c-98ac-759f60d92cbf-maas-api-openshift-ca" pod "28dab9fe-c0a9-465c-98ac-759f60d92cbf" (UID: "28dab9fe-c0a9-465c-98ac-759f60d92cbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:11:25.997640 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:11:25.925629 5463 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.459s" Nov 03 02:11:25.998302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:25.998185 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:11:26.103218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:26.083947 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28dab9fe-c0a9-465c-98ac-759f60d92cbf-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:26.194739 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:26.194605 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28dab9fe-c0a9-465c-98ac-759f60d92cbf-maas-api-xks-ca" pod "28dab9fe-c0a9-465c-98ac-759f60d92cbf" (UID: "28dab9fe-c0a9-465c-98ac-759f60d92cbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:11:26.221657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:26.221620 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 02:11:26.324438 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:26.251432 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h" Nov 03 02:11:26.325253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:26.251453 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h" event={"ID":"28dab9fe-c0a9-465c-98ac-759f60d92cbf","Type":"ContainerDied","Data":"814d79bd0bab23e67ef7dd2ce6b81a8bbaa82384cbbd79acc98553d985f0d544"} Nov 03 02:11:26.326963 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:26.326941 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="814d79bd0bab23e67ef7dd2ce6b81a8bbaa82384cbbd79acc98553d985f0d544" Nov 03 02:11:26.327219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:26.325726 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" podUID="2617013b-bed0-446d-a597-31bb323d1c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 02:11:26.327383 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:26.229598 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28dab9fe-c0a9-465c-98ac-759f60d92cbf-kube-api-access-vt6l5" pod "28dab9fe-c0a9-465c-98ac-759f60d92cbf" (UID: "28dab9fe-c0a9-465c-98ac-759f60d92cbf"). InnerVolumeSpecName "kube-api-access-vt6l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:11:26.331905 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:26.331881 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28dab9fe-c0a9-465c-98ac-759f60d92cbf-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:26.332048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:26.332030 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt6l5\" (UniqueName: \"kubernetes.io/projected/28dab9fe-c0a9-465c-98ac-759f60d92cbf-kube-api-access-vt6l5\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:26.512338 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:26.512291 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 03 02:11:26.537217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:26.524602 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 02:11:26.545216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:26.536313 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 03 02:11:26.557216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:26.543263 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:11:26.689269 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:26.597573 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" podUID="13e969fe-f189-40f2-b929-e462f6d4b86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:26.689269 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:26.650383 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 02:11:26.845218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:26.837975 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 03 02:11:27.015379 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:27.015287 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 02:11:27.015682 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:27.015636 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" podUID="2617013b-bed0-446d-a597-31bb323d1c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 02:11:27.240234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:27.235158 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:27.531222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:27.526469 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 03 02:11:27.558218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:27.548417 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:27.559117 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:27.559055 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:27.653159 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:27.653112 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:27.870329 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:27.870283 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:28.015210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.015100 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 02:11:28.015489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.015443 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" podUID="2617013b-bed0-446d-a597-31bb323d1c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 02:11:28.523216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.516090 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:11:28.529311 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.529269 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:28.560322 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.560264 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:28.596000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.591166 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw"] Nov 03 02:11:28.596000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.592127 5463 cpu_manager.go:414] "containerMap: removing container" podUID="28dab9fe-c0a9-465c-98ac-759f60d92cbf" containerName="revoke-keys" Nov 03 02:11:28.596000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.592149 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28dab9fe-c0a9-465c-98ac-759f60d92cbf" containerName="revoke-keys" Nov 03 02:11:28.600763 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.600478 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="28dab9fe-c0a9-465c-98ac-759f60d92cbf" containerName="revoke-keys" Nov 03 02:11:28.601922 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.601769 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:28.610772 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.610728 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:28.664791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.656006 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:28.704317 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.704278 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw"] Nov 03 02:11:28.848084 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.847981 5463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Nov 03 02:11:28.876866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.876827 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:28.883341 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.883299 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f7d9bb\"/\"distinct-2-w0-f7d9bb-kserve-self-signed-certs\"" Nov 03 02:11:28.884512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.884475 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:28.887152 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.887085 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-tmp-dir\") pod \"distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:28.887391 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.887368 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ba01dba-9d25-480d-b5d1-f51295ed81b7-tls-certs\") pod \"distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:28.887549 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.887532 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-model-cache\") pod \"distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:28.887724 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.887705 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqbnk\" (UniqueName: \"kubernetes.io/projected/1ba01dba-9d25-480d-b5d1-f51295ed81b7-kube-api-access-sqbnk\") pod \"distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:28.887977 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.887959 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-dshm\") pod \"distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:28.888314 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.888291 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-home\") pod \"distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:29.023216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:28.998776 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ba01dba-9d25-480d-b5d1-f51295ed81b7-tls-certs\") pod \"distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:29.095799 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.095751 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-model-cache\") pod \"distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:29.096125 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.096099 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqbnk\" (UniqueName: \"kubernetes.io/projected/1ba01dba-9d25-480d-b5d1-f51295ed81b7-kube-api-access-sqbnk\") pod \"distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:29.138079 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.096418 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-dshm\") pod \"distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:29.138585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.138553 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-home\") pod \"distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:29.140297 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.140275 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-tmp-dir\") pod \"distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:29.140910 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.035297 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ba01dba-9d25-480d-b5d1-f51295ed81b7-tls-certs\") pod \"distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:29.141057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.105553 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-model-cache\") pod \"distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:29.141171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.140081 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-home\") pod \"distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:29.141317 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.025091 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 02:11:29.141469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.141445 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" podUID="2617013b-bed0-446d-a597-31bb323d1c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 02:11:29.246117 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.140693 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-tmp-dir\") pod \"distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:29.330213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.327568 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-dshm\") pod \"distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:29.380181 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.380128 5463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:29.405299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.390994 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:29.549361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.549315 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:29.577267 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.577220 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:29.578253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.578218 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:29.642504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.642467 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:29.660680 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.660591 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:29.686754 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.686716 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqbnk\" (UniqueName: \"kubernetes.io/projected/1ba01dba-9d25-480d-b5d1-f51295ed81b7-kube-api-access-sqbnk\") pod \"distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:29.710226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.707706 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:11:29.745428 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.745393 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:29.752428 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.745605 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:11:29.777544 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.775093 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:11:29.825216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:29.824223 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:11:30.063363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:30.041619 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 02:11:30.063363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:30.041678 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" podUID="2617013b-bed0-446d-a597-31bb323d1c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 02:11:31.017114 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:31.017083 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:31.118354 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:31.118221 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw"] Nov 03 02:11:31.406391 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:31.406348 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" event={"ID":"1ba01dba-9d25-480d-b5d1-f51295ed81b7","Type":"ContainerStarted","Data":"39599de2064b6ff9abcba65459238475edaf70d43135de4189cdb08da3d3d65e"} Nov 03 02:11:31.406594 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:31.406404 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" event={"ID":"1ba01dba-9d25-480d-b5d1-f51295ed81b7","Type":"ContainerStarted","Data":"97693b516a6dff9535d9c52b9e2af9b6d9b62edb6342dd1ce5f368449cc7ca1b"} Nov 03 02:11:31.527889 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:31.527820 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" podStartSLOduration=3.527800242 podStartE2EDuration="3.527800242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:11:31.523903766 +0000 UTC m=+1173.208012922" watchObservedRunningTime="2026-11-03 02:11:31.527800242 +0000 UTC m=+1173.211909399" Nov 03 02:11:31.746549 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:31.746469 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:31.930808 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:31.930774 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:32.119531 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:32.078752 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:11:32.119531 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:32.102358 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" podUID="2617013b-bed0-446d-a597-31bb323d1c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:32.800425 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:32.800392 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:32.802086 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:32.802064 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:11:33.972710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:33.972602 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:34.147669 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:34.147632 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:34.347751 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:34.347658 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:35.523244 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:35.523178 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:37.047567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:37.047527 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:38.052820 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:38.052782 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:38.239306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:38.238023 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:39.049702 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:39.049630 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg"] Nov 03 02:11:39.100039 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:39.100011 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb051bd-dn4zg"] Nov 03 02:11:39.747075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:39.747041 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:39.759893 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:39.759866 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:40.444279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:40.444246 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29ebdadb-dc6c-4cef-952a-04f78c57e88c" path="/var/lib/kubelet/pods/29ebdadb-dc6c-4cef-952a-04f78c57e88c/volumes" Nov 03 02:11:40.487449 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:40.487424 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:41.838276 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:41.838243 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:43.008567 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:11:43.008529 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:11:46.536291 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:46.536245 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:46.834377 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:46.834302 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62"] Nov 03 02:11:46.838602 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:46.838583 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62" Nov 03 02:11:46.891491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:46.891465 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:11:46.891625 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:46.891560 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:11:46.891748 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:46.891724 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62"] Nov 03 02:11:46.891821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:46.891751 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:11:46.891821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:46.891805 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:11:46.983281 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:46.983253 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d4d6abf-5729-4728-b755-64c2231dd9bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ddcb09b-mwm62\" (UID: \"3d4d6abf-5729-4728-b755-64c2231dd9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62" Nov 03 02:11:46.983472 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:46.983378 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xsfr\" (UniqueName: \"kubernetes.io/projected/3d4d6abf-5729-4728-b755-64c2231dd9bf-kube-api-access-8xsfr\") pod \"maas-api-revoke-sub-6ddcb09b-mwm62\" (UID: \"3d4d6abf-5729-4728-b755-64c2231dd9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62" Nov 03 02:11:46.983582 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:46.983566 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d4d6abf-5729-4728-b755-64c2231dd9bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ddcb09b-mwm62\" (UID: \"3d4d6abf-5729-4728-b755-64c2231dd9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62" Nov 03 02:11:47.084236 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:47.084201 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d4d6abf-5729-4728-b755-64c2231dd9bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ddcb09b-mwm62\" (UID: \"3d4d6abf-5729-4728-b755-64c2231dd9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62" Nov 03 02:11:47.084434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:47.084272 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d4d6abf-5729-4728-b755-64c2231dd9bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ddcb09b-mwm62\" (UID: \"3d4d6abf-5729-4728-b755-64c2231dd9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62" Nov 03 02:11:47.084434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:47.084357 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xsfr\" (UniqueName: \"kubernetes.io/projected/3d4d6abf-5729-4728-b755-64c2231dd9bf-kube-api-access-8xsfr\") pod \"maas-api-revoke-sub-6ddcb09b-mwm62\" (UID: \"3d4d6abf-5729-4728-b755-64c2231dd9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62" Nov 03 02:11:47.084866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:47.084845 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d4d6abf-5729-4728-b755-64c2231dd9bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ddcb09b-mwm62\" (UID: \"3d4d6abf-5729-4728-b755-64c2231dd9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62" Nov 03 02:11:47.086688 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:47.086664 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d4d6abf-5729-4728-b755-64c2231dd9bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ddcb09b-mwm62\" (UID: \"3d4d6abf-5729-4728-b755-64c2231dd9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62" Nov 03 02:11:47.142491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:47.142431 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xsfr\" (UniqueName: \"kubernetes.io/projected/3d4d6abf-5729-4728-b755-64c2231dd9bf-kube-api-access-8xsfr\") pod \"maas-api-revoke-sub-6ddcb09b-mwm62\" (UID: \"3d4d6abf-5729-4728-b755-64c2231dd9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62" Nov 03 02:11:47.152252 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:47.152217 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62" Nov 03 02:11:47.538787 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:11:47.538737 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4d6abf_5729_4728_b755_64c2231dd9bf.slice/crio-5ae0ebf24f7d23054465a2c1ea006e2d53a3caacc7969200b84a5244f1fa6903 WatchSource:0}: Error finding container 5ae0ebf24f7d23054465a2c1ea006e2d53a3caacc7969200b84a5244f1fa6903: Status 404 returned error can't find the container with id 5ae0ebf24f7d23054465a2c1ea006e2d53a3caacc7969200b84a5244f1fa6903 Nov 03 02:11:47.544622 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:47.544596 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62"] Nov 03 02:11:47.916531 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:47.916506 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:48.539886 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:48.539850 5463 generic.go:356] "Generic (PLEG): container finished" podID="3d4d6abf-5729-4728-b755-64c2231dd9bf" containerID="dcdf053bfeba3ea84b2f1099f8f883bb39e4f67de28ccf31d9bc544f0ec9f4e5" exitCode=0 Nov 03 02:11:48.540365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:48.539937 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62" event={"ID":"3d4d6abf-5729-4728-b755-64c2231dd9bf","Type":"ContainerDied","Data":"dcdf053bfeba3ea84b2f1099f8f883bb39e4f67de28ccf31d9bc544f0ec9f4e5"} Nov 03 02:11:48.540365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:48.539989 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62" event={"ID":"3d4d6abf-5729-4728-b755-64c2231dd9bf","Type":"ContainerStarted","Data":"5ae0ebf24f7d23054465a2c1ea006e2d53a3caacc7969200b84a5244f1fa6903"} Nov 03 02:11:49.709112 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:49.709091 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62" Nov 03 02:11:49.819100 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:49.819026 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4d6abf-5729-4728-b755-64c2231dd9bf-kube-api-access-8xsfr\" (UniqueName: \"kubernetes.io/projected/3d4d6abf-5729-4728-b755-64c2231dd9bf-kube-api-access-8xsfr\") pod \"3d4d6abf-5729-4728-b755-64c2231dd9bf\" (UID: \"3d4d6abf-5729-4728-b755-64c2231dd9bf\") " Nov 03 02:11:49.819251 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:49.819123 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d4d6abf-5729-4728-b755-64c2231dd9bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d4d6abf-5729-4728-b755-64c2231dd9bf-maas-api-xks-ca\") pod \"3d4d6abf-5729-4728-b755-64c2231dd9bf\" (UID: \"3d4d6abf-5729-4728-b755-64c2231dd9bf\") " Nov 03 02:11:49.819251 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:49.819211 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d4d6abf-5729-4728-b755-64c2231dd9bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d4d6abf-5729-4728-b755-64c2231dd9bf-maas-api-openshift-ca\") pod \"3d4d6abf-5729-4728-b755-64c2231dd9bf\" (UID: \"3d4d6abf-5729-4728-b755-64c2231dd9bf\") " Nov 03 02:11:49.819568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:49.819535 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4d6abf-5729-4728-b755-64c2231dd9bf-maas-api-openshift-ca" pod "3d4d6abf-5729-4728-b755-64c2231dd9bf" (UID: "3d4d6abf-5729-4728-b755-64c2231dd9bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:11:49.819825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:49.819786 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d4d6abf-5729-4728-b755-64c2231dd9bf-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:49.821870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:49.821829 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d4d6abf-5729-4728-b755-64c2231dd9bf-maas-api-xks-ca" pod "3d4d6abf-5729-4728-b755-64c2231dd9bf" (UID: "3d4d6abf-5729-4728-b755-64c2231dd9bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:11:49.822541 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:49.822519 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4d6abf-5729-4728-b755-64c2231dd9bf-kube-api-access-8xsfr" pod "3d4d6abf-5729-4728-b755-64c2231dd9bf" (UID: "3d4d6abf-5729-4728-b755-64c2231dd9bf"). InnerVolumeSpecName "kube-api-access-8xsfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:11:49.921152 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:49.921121 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d4d6abf-5729-4728-b755-64c2231dd9bf-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:49.921312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:49.921155 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xsfr\" (UniqueName: \"kubernetes.io/projected/3d4d6abf-5729-4728-b755-64c2231dd9bf-kube-api-access-8xsfr\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:50.558947 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:50.558918 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62" Nov 03 02:11:50.558947 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:50.558934 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62" event={"ID":"3d4d6abf-5729-4728-b755-64c2231dd9bf","Type":"ContainerDied","Data":"5ae0ebf24f7d23054465a2c1ea006e2d53a3caacc7969200b84a5244f1fa6903"} Nov 03 02:11:50.559137 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:50.558974 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ae0ebf24f7d23054465a2c1ea006e2d53a3caacc7969200b84a5244f1fa6903" Nov 03 02:11:51.341267 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:51.341229 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:56.024515 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:56.024474 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:56.459210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:56.458936 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:56.628216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:56.627652 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:57.463321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:57.463272 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:58.009962 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:58.009819 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:58.193213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:58.192291 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 03 02:11:58.269416 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:58.269311 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:11:58.269945 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:58.269911 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" podUID="1e5aa4d8-7e07-4efe-96ef-93b35748423e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:58.455360 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:58.447424 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:58.482216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:58.481727 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" podUID="1ba01dba-9d25-480d-b5d1-f51295ed81b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:58.537298 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:58.536534 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:58.565227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:58.564651 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:58.862265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:58.859285 5463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Nov 03 02:11:59.209215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:59.208795 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:59.213210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:59.209497 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:11:59.532730 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:59.532639 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" podUID="1ba01dba-9d25-480d-b5d1-f51295ed81b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 03 02:11:59.551992 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:59.533757 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:11:59.553343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:59.540504 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:59.554424 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:59.540559 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:59.554858 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:59.554839 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:11:59.555545 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:59.540712 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:59.555960 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:59.555942 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:11:59.580215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:59.577364 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:11:59.580215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:59.577696 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:11:59.682558 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:59.682526 5463 scope.go:122] "RemoveContainer" containerID="e8aaa70c6ee5d24ef939d3a46b7842e79c7300633a420787ffa771c0692a74e2" Nov 03 02:11:59.934215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:11:59.909602 5463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:00.219095 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:00.219002 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:00.408032 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:00.407997 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:00.495091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:00.495049 5463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:00.749272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:00.745569 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:00.840394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:00.790871 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" podUID="1ba01dba-9d25-480d-b5d1-f51295ed81b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 03 02:12:00.840394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:00.791232 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:00.840394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:00.791476 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 02:12:00.863247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:00.841287 5463 scope.go:122] "RemoveContainer" containerID="91208b156a2e1d95fa6bfbbe3a094da993ad1321125ebe51b2502425b55a7405" Nov 03 02:12:00.962273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:00.952939 5463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" podUID="1ba01dba-9d25-480d-b5d1-f51295ed81b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/health\": context deadline exceeded" Nov 03 02:12:01.080051 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.050661 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:01.087412 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.087377 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qxq6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:12:01.087670 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.087637 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" podUID="0e38d821-9e7d-46a4-93a1-4b4285b1de35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:01.221573 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.221542 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gcscd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:12:01.230889 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.230853 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" podUID="5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:01.296637 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.296370 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:01.342272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.341294 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jlvds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:12:01.342272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.341360 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" podUID="6fc604c8-be34-40a7-9fb8-a07424233eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:01.535216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.514816 5463 scope.go:122] "RemoveContainer" containerID="3d8f8c7c7f7d5f4946386d2350cddd3378e71b32a6e8265ec328f1399f0aa85d" Nov 03 02:12:01.535216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.534620 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:01.557255 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:12:01.545476 5463 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.077s" Nov 03 02:12:01.557255 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.545558 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:12:01.557255 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.545657 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:12:01.557255 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.545692 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:12:01.641426 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.639394 5463 scope.go:122] "RemoveContainer" containerID="f0a8d53788bf459a5e7830e8adeddc276e02e2ee2a7ed8099fd8313ffe71398a" Nov 03 02:12:01.736719 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.736664 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:01.854214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.817376 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:01.854214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.817738 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" podUID="1ba01dba-9d25-480d-b5d1-f51295ed81b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 03 02:12:01.902670 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.859273 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:01.916845 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.913591 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78"] Nov 03 02:12:01.916845 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.914695 5463 cpu_manager.go:414] "containerMap: removing container" podUID="3d4d6abf-5729-4728-b755-64c2231dd9bf" containerName="revoke-keys" Nov 03 02:12:01.916845 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.914716 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4d6abf-5729-4728-b755-64c2231dd9bf" containerName="revoke-keys" Nov 03 02:12:01.916845 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:01.914867 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4d6abf-5729-4728-b755-64c2231dd9bf" containerName="revoke-keys" Nov 03 02:12:02.147060 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.146617 5463 scope.go:122] "RemoveContainer" containerID="95aec74faf3f39e2bebf5261b76aabf6c934a2d99fc2749bda1c02e3bc133379" Nov 03 02:12:02.174571 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.157127 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78"] Nov 03 02:12:02.174571 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.157295 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78" Nov 03 02:12:02.217014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.216685 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:12:02.217014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.216768 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:12:02.217014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.216858 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:12:02.217014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.216947 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:12:02.288384 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.272403 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbcf599a-af8c-494c-82d4-79ee96ea65c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fcafcbb-svc78\" (UID: \"bbcf599a-af8c-494c-82d4-79ee96ea65c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78" Nov 03 02:12:02.288384 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.272474 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf5m9\" (UniqueName: \"kubernetes.io/projected/bbcf599a-af8c-494c-82d4-79ee96ea65c1-kube-api-access-kf5m9\") pod \"maas-api-revoke-sub-2fcafcbb-svc78\" (UID: \"bbcf599a-af8c-494c-82d4-79ee96ea65c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78" Nov 03 02:12:02.288384 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.272664 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbcf599a-af8c-494c-82d4-79ee96ea65c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fcafcbb-svc78\" (UID: \"bbcf599a-af8c-494c-82d4-79ee96ea65c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78" Nov 03 02:12:02.311147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.296868 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:02.393019 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.392987 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf5m9\" (UniqueName: \"kubernetes.io/projected/bbcf599a-af8c-494c-82d4-79ee96ea65c1-kube-api-access-kf5m9\") pod \"maas-api-revoke-sub-2fcafcbb-svc78\" (UID: \"bbcf599a-af8c-494c-82d4-79ee96ea65c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78" Nov 03 02:12:02.393443 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.393419 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbcf599a-af8c-494c-82d4-79ee96ea65c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fcafcbb-svc78\" (UID: \"bbcf599a-af8c-494c-82d4-79ee96ea65c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78" Nov 03 02:12:02.393756 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.393735 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbcf599a-af8c-494c-82d4-79ee96ea65c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fcafcbb-svc78\" (UID: \"bbcf599a-af8c-494c-82d4-79ee96ea65c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78" Nov 03 02:12:02.400820 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.400796 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbcf599a-af8c-494c-82d4-79ee96ea65c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fcafcbb-svc78\" (UID: \"bbcf599a-af8c-494c-82d4-79ee96ea65c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78" Nov 03 02:12:02.414727 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.414702 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 02:12:02.477938 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.477840 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbcf599a-af8c-494c-82d4-79ee96ea65c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fcafcbb-svc78\" (UID: \"bbcf599a-af8c-494c-82d4-79ee96ea65c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78" Nov 03 02:12:02.501984 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.501958 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:12:02.511273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.511247 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:12:02.536667 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.536628 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:02.537570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.537545 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:12:02.811222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.803704 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:02.857576 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.857537 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" podUID="1ba01dba-9d25-480d-b5d1-f51295ed81b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:02.897597 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.859383 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:02.897597 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.859439 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:02.897597 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.897134 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf5m9\" (UniqueName: \"kubernetes.io/projected/bbcf599a-af8c-494c-82d4-79ee96ea65c1-kube-api-access-kf5m9\") pod \"maas-api-revoke-sub-2fcafcbb-svc78\" (UID: \"bbcf599a-af8c-494c-82d4-79ee96ea65c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78" Nov 03 02:12:02.930560 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:02.907823 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 02:12:03.027214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:03.026394 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78" Nov 03 02:12:03.103470 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:03.102101 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 02:12:03.123543 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:03.111737 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 02:12:03.199430 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:03.189598 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 02:12:03.722680 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:03.722647 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78"] Nov 03 02:12:03.725518 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:12:03.725486 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbcf599a_af8c_494c_82d4_79ee96ea65c1.slice/crio-3b80e032f96b269ce07459bf024070b36662a824e57fdaae5fb726151fe9b691 WatchSource:0}: Error finding container 3b80e032f96b269ce07459bf024070b36662a824e57fdaae5fb726151fe9b691: Status 404 returned error can't find the container with id 3b80e032f96b269ce07459bf024070b36662a824e57fdaae5fb726151fe9b691 Nov 03 02:12:03.748822 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:03.748795 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78" event={"ID":"bbcf599a-af8c-494c-82d4-79ee96ea65c1","Type":"ContainerStarted","Data":"3b80e032f96b269ce07459bf024070b36662a824e57fdaae5fb726151fe9b691"} Nov 03 02:12:04.757095 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:04.757048 5463 generic.go:356] "Generic (PLEG): container finished" podID="bbcf599a-af8c-494c-82d4-79ee96ea65c1" containerID="ebaf64202acc66878448e62ad1da0625d9fde336944f8f5633b4e7e3aa8112b0" exitCode=0 Nov 03 02:12:04.757601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:04.757131 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78" event={"ID":"bbcf599a-af8c-494c-82d4-79ee96ea65c1","Type":"ContainerDied","Data":"ebaf64202acc66878448e62ad1da0625d9fde336944f8f5633b4e7e3aa8112b0"} Nov 03 02:12:05.971303 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:05.971278 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78" Nov 03 02:12:06.028487 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:06.028404 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbcf599a-af8c-494c-82d4-79ee96ea65c1-kube-api-access-kf5m9\" (UniqueName: \"kubernetes.io/projected/bbcf599a-af8c-494c-82d4-79ee96ea65c1-kube-api-access-kf5m9\") pod \"bbcf599a-af8c-494c-82d4-79ee96ea65c1\" (UID: \"bbcf599a-af8c-494c-82d4-79ee96ea65c1\") " Nov 03 02:12:06.028487 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:06.028442 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbcf599a-af8c-494c-82d4-79ee96ea65c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbcf599a-af8c-494c-82d4-79ee96ea65c1-maas-api-xks-ca\") pod \"bbcf599a-af8c-494c-82d4-79ee96ea65c1\" (UID: \"bbcf599a-af8c-494c-82d4-79ee96ea65c1\") " Nov 03 02:12:06.028718 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:06.028623 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbcf599a-af8c-494c-82d4-79ee96ea65c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbcf599a-af8c-494c-82d4-79ee96ea65c1-maas-api-openshift-ca\") pod \"bbcf599a-af8c-494c-82d4-79ee96ea65c1\" (UID: \"bbcf599a-af8c-494c-82d4-79ee96ea65c1\") " Nov 03 02:12:06.029101 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:06.029062 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbcf599a-af8c-494c-82d4-79ee96ea65c1-maas-api-openshift-ca" pod "bbcf599a-af8c-494c-82d4-79ee96ea65c1" (UID: "bbcf599a-af8c-494c-82d4-79ee96ea65c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:12:06.029261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:06.029225 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbcf599a-af8c-494c-82d4-79ee96ea65c1-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:12:06.030637 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:06.030608 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbcf599a-af8c-494c-82d4-79ee96ea65c1-maas-api-xks-ca" pod "bbcf599a-af8c-494c-82d4-79ee96ea65c1" (UID: "bbcf599a-af8c-494c-82d4-79ee96ea65c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:12:06.030851 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:06.030826 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbcf599a-af8c-494c-82d4-79ee96ea65c1-kube-api-access-kf5m9" pod "bbcf599a-af8c-494c-82d4-79ee96ea65c1" (UID: "bbcf599a-af8c-494c-82d4-79ee96ea65c1"). InnerVolumeSpecName "kube-api-access-kf5m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:12:06.130972 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:06.130935 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf5m9\" (UniqueName: \"kubernetes.io/projected/bbcf599a-af8c-494c-82d4-79ee96ea65c1-kube-api-access-kf5m9\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:12:06.130972 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:06.130966 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbcf599a-af8c-494c-82d4-79ee96ea65c1-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:12:06.340406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:06.340316 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:06.490353 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:06.490325 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:06.645416 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:06.645385 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:06.778131 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:06.778101 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78" Nov 03 02:12:06.778357 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:06.778132 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78" event={"ID":"bbcf599a-af8c-494c-82d4-79ee96ea65c1","Type":"ContainerDied","Data":"3b80e032f96b269ce07459bf024070b36662a824e57fdaae5fb726151fe9b691"} Nov 03 02:12:06.778357 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:06.778217 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b80e032f96b269ce07459bf024070b36662a824e57fdaae5fb726151fe9b691" Nov 03 02:12:10.102216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:10.100692 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:11.212663 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:11.212614 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:13.013519 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:12:13.013469 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:12:13.340715 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.340638 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p"] Nov 03 02:12:13.341295 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.341281 5463 cpu_manager.go:414] "containerMap: removing container" podUID="bbcf599a-af8c-494c-82d4-79ee96ea65c1" containerName="revoke-keys" Nov 03 02:12:13.341347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.341296 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbcf599a-af8c-494c-82d4-79ee96ea65c1" containerName="revoke-keys" Nov 03 02:12:13.341402 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.341393 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbcf599a-af8c-494c-82d4-79ee96ea65c1" containerName="revoke-keys" Nov 03 02:12:13.344671 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.344652 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.425704 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.425668 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p"] Nov 03 02:12:13.439844 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.439808 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.440003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.439952 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.440003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.439984 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.440141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.440008 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.440141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.440056 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shks9\" (UniqueName: \"kubernetes.io/projected/547c8528-2b07-475d-a191-f0de8a63a4b9-kube-api-access-shks9\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.440141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.440104 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.440141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.440128 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.440413 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.440147 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/547c8528-2b07-475d-a191-f0de8a63a4b9-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.440413 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.440182 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.541861 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.541811 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.541861 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.541850 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.542145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.542054 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.542226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.542171 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shks9\" (UniqueName: \"kubernetes.io/projected/547c8528-2b07-475d-a191-f0de8a63a4b9-kube-api-access-shks9\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.542363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.542340 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.542576 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.542556 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.542749 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.542718 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/547c8528-2b07-475d-a191-f0de8a63a4b9-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.542749 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.542420 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.542922 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.542337 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.542922 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.542820 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.542922 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.542825 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.542922 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.542875 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.543148 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.543091 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.543472 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.543450 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/547c8528-2b07-475d-a191-f0de8a63a4b9-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.544737 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.544706 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.545325 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.545308 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.602084 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.602028 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.602365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.602338 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shks9\" (UniqueName: \"kubernetes.io/projected/547c8528-2b07-475d-a191-f0de8a63a4b9-kube-api-access-shks9\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:13.660774 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:13.660741 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:14.087415 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:12:14.087371 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547c8528_2b07_475d_a191_f0de8a63a4b9.slice/crio-8f5398e2385aa7bcd48a7823b930eef82410af0f7387dfe50363741a9b09ae80 WatchSource:0}: Error finding container 8f5398e2385aa7bcd48a7823b930eef82410af0f7387dfe50363741a9b09ae80: Status 404 returned error can't find the container with id 8f5398e2385aa7bcd48a7823b930eef82410af0f7387dfe50363741a9b09ae80 Nov 03 02:12:14.090487 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:14.090442 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:12:14.090590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:14.090559 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:12:14.090629 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:14.090609 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:12:14.092184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:14.092164 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p"] Nov 03 02:12:14.110505 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:14.105317 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:14.259840 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:14.259802 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:15.238530 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:15.238486 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" event={"ID":"547c8528-2b07-475d-a191-f0de8a63a4b9","Type":"ContainerStarted","Data":"4ff036340ad598d738328bc2046c567f0dde8508f0aa04e6d148a235e8a3bb2c"} Nov 03 02:12:15.242360 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:15.242318 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" event={"ID":"547c8528-2b07-475d-a191-f0de8a63a4b9","Type":"ContainerStarted","Data":"8f5398e2385aa7bcd48a7823b930eef82410af0f7387dfe50363741a9b09ae80"} Nov 03 02:12:15.411318 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:15.411251 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" podStartSLOduration=2.411237832 podStartE2EDuration="2.411237832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:12:15.408677299 +0000 UTC m=+1217.092786453" watchObservedRunningTime="2026-11-03 02:12:15.411237832 +0000 UTC m=+1217.095346989" Nov 03 02:12:15.671102 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:15.671069 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:15.672368 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:15.672325 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 02:12:15.672504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:15.672417 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" podUID="547c8528-2b07-475d-a191-f0de8a63a4b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 02:12:16.173144 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:16.173112 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:16.668603 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:16.668566 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 02:12:16.669252 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:16.669219 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" podUID="547c8528-2b07-475d-a191-f0de8a63a4b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 02:12:17.665520 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:17.665471 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 02:12:17.665709 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:17.665541 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" podUID="547c8528-2b07-475d-a191-f0de8a63a4b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 02:12:18.672218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:18.669302 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 02:12:18.672218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:18.669363 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" podUID="547c8528-2b07-475d-a191-f0de8a63a4b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 02:12:19.665216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:19.661356 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 02:12:19.665216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:19.661429 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" podUID="547c8528-2b07-475d-a191-f0de8a63a4b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 02:12:20.661328 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:20.661285 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 02:12:20.661721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:20.661368 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" podUID="547c8528-2b07-475d-a191-f0de8a63a4b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 02:12:20.914640 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:20.914559 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:21.694508 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:21.694473 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:21.696033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:21.696001 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:21.697150 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:21.697124 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:12:25.684344 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:25.684308 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:30.419933 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:30.419887 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:32.823306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:32.823260 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 02:12:33.042992 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:33.042941 5463 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:33.106350 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:33.106252 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:12:35.329504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:35.329458 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:40.029026 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:40.028984 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:43.010497 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:12:43.010459 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:12:44.742946 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:44.742912 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:49.831120 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:49.831088 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:50.311351 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:50.311313 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:54.049544 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.049500 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:54.245923 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.245881 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:54.559110 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.559076 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:54.650206 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.650164 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n"] Nov 03 02:12:54.654715 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.654692 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.760987 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.760954 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:54.769210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.766662 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n"] Nov 03 02:12:54.796222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.794471 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7f332d-openshift-default-dockercfg-jch7q\"" Nov 03 02:12:54.847825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.847719 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t4vz\" (UniqueName: \"kubernetes.io/projected/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-kube-api-access-2t4vz\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.848015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.847831 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-token\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.848015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.847878 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-credential-socket\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.848015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.847972 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istiod-ca-cert\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.848222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.848032 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-podinfo\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.848222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.848114 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-workload-socket\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.848339 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.848234 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-workload-certs\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.848339 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.848268 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-envoy\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.848452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.848351 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-data\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.953768 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.953729 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-data\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.955033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.954036 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t4vz\" (UniqueName: \"kubernetes.io/projected/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-kube-api-access-2t4vz\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.955033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.954134 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-token\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.955033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.954176 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-credential-socket\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.955033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.954310 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istiod-ca-cert\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.955033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.954367 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-podinfo\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.955033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.954440 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-workload-socket\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.955033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.954527 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-workload-certs\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.955033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.954562 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-envoy\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.958033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.957400 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-workload-socket\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.958033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.957673 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-workload-certs\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.958693 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.958674 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istiod-ca-cert\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.959240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.959217 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-data\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.959624 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.959605 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-credential-socket\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.962628 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.962599 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-podinfo\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:54.993214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:54.981539 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-envoy\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:55.013394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:55.013328 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-token\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:55.015889 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:55.015858 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t4vz\" (UniqueName: \"kubernetes.io/projected/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-kube-api-access-2t4vz\") pod \"e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:55.072158 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:55.072121 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:55.271490 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:55.271450 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:55.708717 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:55.708686 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n"] Nov 03 02:12:55.739818 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:12:55.739790 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec04276_3f4d_4bdf_96bf_37f93fd0913e.slice/crio-4070552af0ef31f6818248a107c83fe8fcca9d370c960a37910b2724cfb6b6a3 WatchSource:0}: Error finding container 4070552af0ef31f6818248a107c83fe8fcca9d370c960a37910b2724cfb6b6a3: Status 404 returned error can't find the container with id 4070552af0ef31f6818248a107c83fe8fcca9d370c960a37910b2724cfb6b6a3 Nov 03 02:12:55.742258 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:55.742223 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:12:55.742347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:55.742291 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:12:55.742347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:55.742324 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:12:56.686445 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:56.686407 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" event={"ID":"9ec04276-3f4d-4bdf-96bf-37f93fd0913e","Type":"ContainerStarted","Data":"bdc92936e7c5552b0cdaa645cabf9cb7a0f50f330262a1963352820dfa82ae06"} Nov 03 02:12:56.686935 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:56.686501 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" event={"ID":"9ec04276-3f4d-4bdf-96bf-37f93fd0913e","Type":"ContainerStarted","Data":"4070552af0ef31f6818248a107c83fe8fcca9d370c960a37910b2724cfb6b6a3"} Nov 03 02:12:56.855380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:56.855331 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" podStartSLOduration=2.855317825 podStartE2EDuration="2.855317825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:12:56.852256633 +0000 UTC m=+1258.536365790" watchObservedRunningTime="2026-11-03 02:12:56.855317825 +0000 UTC m=+1258.539427052" Nov 03 02:12:57.272282 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:57.272248 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:57.277151 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:57.277128 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:57.693966 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:57.693933 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:57.694885 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:57.694865 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:12:59.435212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:59.435168 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:59.613363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:59.613329 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:59.785643 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:12:59.785560 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:02.841921 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:02.841883 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:03.039870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.039823 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:03.239832 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.239796 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b"] Nov 03 02:13:03.244259 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.244241 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b" Nov 03 02:13:03.281132 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.281099 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjfvt\" (UniqueName: \"kubernetes.io/projected/490ca339-509d-4f76-b2b7-6e6f57be24c7-kube-api-access-xjfvt\") pod \"payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b\" (UID: \"490ca339-509d-4f76-b2b7-6e6f57be24c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b" Nov 03 02:13:03.281415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.281392 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490ca339-509d-4f76-b2b7-6e6f57be24c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b\" (UID: \"490ca339-509d-4f76-b2b7-6e6f57be24c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b" Nov 03 02:13:03.298220 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.298162 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b"] Nov 03 02:13:03.325286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.325261 5463 scope.go:122] "RemoveContainer" containerID="a8754b43bb01b6d2f3693be891a1973a837f18b95dad122d252bf43ea74a99ad" Nov 03 02:13:03.342182 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.341982 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7f332d\"" Nov 03 02:13:03.342182 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.342151 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-7f332d-dockercfg-7hwkn\"" Nov 03 02:13:03.393246 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.391161 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjfvt\" (UniqueName: \"kubernetes.io/projected/490ca339-509d-4f76-b2b7-6e6f57be24c7-kube-api-access-xjfvt\") pod \"payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b\" (UID: \"490ca339-509d-4f76-b2b7-6e6f57be24c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b" Nov 03 02:13:03.393246 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.391473 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490ca339-509d-4f76-b2b7-6e6f57be24c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b\" (UID: \"490ca339-509d-4f76-b2b7-6e6f57be24c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b" Nov 03 02:13:03.393246 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.392510 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490ca339-509d-4f76-b2b7-6e6f57be24c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b\" (UID: \"490ca339-509d-4f76-b2b7-6e6f57be24c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b" Nov 03 02:13:03.468274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.461289 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck"] Nov 03 02:13:03.468274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.467737 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjfvt\" (UniqueName: \"kubernetes.io/projected/490ca339-509d-4f76-b2b7-6e6f57be24c7-kube-api-access-xjfvt\") pod \"payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b\" (UID: \"490ca339-509d-4f76-b2b7-6e6f57be24c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b" Nov 03 02:13:03.565491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.539623 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck"] Nov 03 02:13:03.565491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.539752 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck" Nov 03 02:13:03.565491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.559488 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b" Nov 03 02:13:03.599265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.594645 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ssbx\" (UniqueName: \"kubernetes.io/projected/8bd6ab07-9430-466d-9162-8e89ace4f015-kube-api-access-9ssbx\") pod \"payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck\" (UID: \"8bd6ab07-9430-466d-9162-8e89ace4f015\") " pod="openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck" Nov 03 02:13:03.599265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.594689 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd6ab07-9430-466d-9162-8e89ace4f015-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck\" (UID: \"8bd6ab07-9430-466d-9162-8e89ace4f015\") " pod="openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck" Nov 03 02:13:03.695772 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.695738 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ssbx\" (UniqueName: \"kubernetes.io/projected/8bd6ab07-9430-466d-9162-8e89ace4f015-kube-api-access-9ssbx\") pod \"payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck\" (UID: \"8bd6ab07-9430-466d-9162-8e89ace4f015\") " pod="openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck" Nov 03 02:13:03.695909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.695790 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd6ab07-9430-466d-9162-8e89ace4f015-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck\" (UID: \"8bd6ab07-9430-466d-9162-8e89ace4f015\") " pod="openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck" Nov 03 02:13:03.696811 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.696786 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd6ab07-9430-466d-9162-8e89ace4f015-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck\" (UID: \"8bd6ab07-9430-466d-9162-8e89ace4f015\") " pod="openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck" Nov 03 02:13:03.755605 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.755287 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ssbx\" (UniqueName: \"kubernetes.io/projected/8bd6ab07-9430-466d-9162-8e89ace4f015-kube-api-access-9ssbx\") pod \"payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck\" (UID: \"8bd6ab07-9430-466d-9162-8e89ace4f015\") " pod="openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck" Nov 03 02:13:03.895018 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.894988 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck" Nov 03 02:13:03.992495 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:03.992466 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b"] Nov 03 02:13:04.286481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:04.286453 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck"] Nov 03 02:13:04.430453 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:04.430422 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:04.749954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:04.749914 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b" event={"ID":"490ca339-509d-4f76-b2b7-6e6f57be24c7","Type":"ContainerStarted","Data":"7de0b497e3f7eab0e36890ea96e067bc27cf75dc16c7fad9fe10c42cf83fd6ba"} Nov 03 02:13:04.750154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:04.749995 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b" Nov 03 02:13:04.750154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:04.750021 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b" event={"ID":"490ca339-509d-4f76-b2b7-6e6f57be24c7","Type":"ContainerStarted","Data":"b1c5ffe54722e1b16d3e1a2d05de96fb616e03e7382f571b6e12ce9850a7ad91"} Nov 03 02:13:04.751773 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:04.751744 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck" event={"ID":"8bd6ab07-9430-466d-9162-8e89ace4f015","Type":"ContainerStarted","Data":"f95354f7969a841134adb8f6b0068362d88f27e2254adf4fc33dad10febeb0ea"} Nov 03 02:13:04.751893 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:04.751795 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck" event={"ID":"8bd6ab07-9430-466d-9162-8e89ace4f015","Type":"ContainerStarted","Data":"d706282e4e286d7584ea345c309fda4cce313c347e9bf738fdc2b80729d7d343"} Nov 03 02:13:04.751893 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:04.751848 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck" Nov 03 02:13:04.869273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:04.869214 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b" podStartSLOduration=1.86918023 podStartE2EDuration="1.86918023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:13:04.863879843 +0000 UTC m=+1266.547989000" watchObservedRunningTime="2026-11-03 02:13:04.86918023 +0000 UTC m=+1266.553289386" Nov 03 02:13:09.124551 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:09.124471 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:10.413680 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:10.413634 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:10.943934 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:10.943897 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:13.008505 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:13:13.008472 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:13:13.837655 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:13.837590 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:14.816817 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:14.816779 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:14.959394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:14.959365 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:15.763007 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:15.762976 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b" Nov 03 02:13:15.763216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:15.763070 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck" Nov 03 02:13:15.879347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:15.878919 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck" podStartSLOduration=12.878903033 podStartE2EDuration="12.878903033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:13:04.987231795 +0000 UTC m=+1266.671340954" watchObservedRunningTime="2026-11-03 02:13:15.878903033 +0000 UTC m=+1277.563012194" Nov 03 02:13:16.725692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:16.725606 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:18.528914 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:18.528839 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:23.338391 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:23.338355 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:27.214608 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:27.214577 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:28.057753 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:28.057697 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:31.162371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:31.162324 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gcscd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:13:31.162762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:31.162407 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" podUID="5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:13:32.966224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:32.966162 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:33.129020 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:33.128987 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:33.283629 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:33.283543 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:33.433788 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:33.433755 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:33.895714 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:33.895679 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m"] Nov 03 02:13:33.900277 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:33.900254 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:33.929762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:33.929727 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-data\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:33.929920 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:33.929778 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-podinfo\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:33.929920 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:33.929854 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztgz8\" (UniqueName: \"kubernetes.io/projected/92398fe7-87d2-4c4f-8108-31a05b23c521-kube-api-access-ztgz8\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:33.930072 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:33.930053 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92398fe7-87d2-4c4f-8108-31a05b23c521-istiod-ca-cert\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:33.930133 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:33.930092 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-workload-socket\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:33.930243 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:33.930228 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-envoy\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:33.930331 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:33.930318 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-workload-certs\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:33.930394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:33.930381 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-credential-socket\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:33.930461 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:33.930414 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-token\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:33.960440 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:33.960406 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m"] Nov 03 02:13:33.998107 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:33.998074 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:34.025254 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.025228 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a772c3-openshift-default-dockercfg-nfwkh\"" Nov 03 02:13:34.032108 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.032047 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-credential-socket\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:34.032108 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.032102 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-token\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:34.032348 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.032225 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-data\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:34.032348 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.032266 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-podinfo\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:34.032348 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.032288 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztgz8\" (UniqueName: \"kubernetes.io/projected/92398fe7-87d2-4c4f-8108-31a05b23c521-kube-api-access-ztgz8\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:34.032514 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.032377 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92398fe7-87d2-4c4f-8108-31a05b23c521-istiod-ca-cert\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:34.032514 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.032412 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-workload-socket\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:34.032692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.032671 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-envoy\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:34.033303 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.032967 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-workload-certs\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:34.033433 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.033342 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-workload-certs\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:34.033505 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.033433 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-credential-socket\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:34.033557 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.033510 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-data\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:34.033557 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.033516 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-workload-socket\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:34.033638 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.033518 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92398fe7-87d2-4c4f-8108-31a05b23c521-istiod-ca-cert\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:34.035362 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.035333 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-envoy\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:34.035460 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.035366 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-podinfo\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:34.090035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.090006 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztgz8\" (UniqueName: \"kubernetes.io/projected/92398fe7-87d2-4c4f-8108-31a05b23c521-kube-api-access-ztgz8\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:34.090322 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.090296 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-token\") pod \"e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:34.173770 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.173653 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:34.221699 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.221661 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:34.776139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.776103 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m"] Nov 03 02:13:34.804681 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:13:34.804656 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92398fe7_87d2_4c4f_8108_31a05b23c521.slice/crio-6d5e8a541b64812415a74b2e94652be417e1bb96046417aba043d74a665c7ead WatchSource:0}: Error finding container 6d5e8a541b64812415a74b2e94652be417e1bb96046417aba043d74a665c7ead: Status 404 returned error can't find the container with id 6d5e8a541b64812415a74b2e94652be417e1bb96046417aba043d74a665c7ead Nov 03 02:13:34.807185 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.807148 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:13:34.807318 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.807297 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:13:34.807392 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:34.807349 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:13:35.068433 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:35.068333 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" event={"ID":"92398fe7-87d2-4c4f-8108-31a05b23c521","Type":"ContainerStarted","Data":"346e364d7f86c01a662ed57bcf73243a146e9f259580707498be7c99a366ee8a"} Nov 03 02:13:35.068433 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:35.068405 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" event={"ID":"92398fe7-87d2-4c4f-8108-31a05b23c521","Type":"ContainerStarted","Data":"6d5e8a541b64812415a74b2e94652be417e1bb96046417aba043d74a665c7ead"} Nov 03 02:13:35.190532 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:35.190474 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" podStartSLOduration=2.190456769 podStartE2EDuration="2.190456769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:13:35.184392075 +0000 UTC m=+1296.868501232" watchObservedRunningTime="2026-11-03 02:13:35.190456769 +0000 UTC m=+1296.874565925" Nov 03 02:13:35.224041 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:35.224000 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:35.229722 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:35.229695 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:36.075815 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:36.075788 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:36.076637 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:36.076619 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:13:38.149854 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:38.149745 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:38.386050 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:38.386014 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:38.548430 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:38.548332 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:38.705823 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:38.705793 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:40.564208 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:40.564161 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:41.597171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:41.597128 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:41.795158 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:41.795119 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:42.194027 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:42.193988 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb"] Nov 03 02:13:42.228291 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:42.225998 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb" Nov 03 02:13:42.293566 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:42.293534 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb"] Nov 03 02:13:42.331560 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:42.329878 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a772c3\"" Nov 03 02:13:42.331560 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:42.330043 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a772c3-dockercfg-vx7pt\"" Nov 03 02:13:42.373651 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:42.373606 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5xqv\" (UniqueName: \"kubernetes.io/projected/f4dc3ccd-f2c5-40f1-b834-f2a064032311-kube-api-access-x5xqv\") pod \"payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb\" (UID: \"f4dc3ccd-f2c5-40f1-b834-f2a064032311\") " pod="openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb" Nov 03 02:13:42.373825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:42.373747 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc3ccd-f2c5-40f1-b834-f2a064032311-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb\" (UID: \"f4dc3ccd-f2c5-40f1-b834-f2a064032311\") " pod="openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb" Nov 03 02:13:42.475613 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:42.475517 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5xqv\" (UniqueName: \"kubernetes.io/projected/f4dc3ccd-f2c5-40f1-b834-f2a064032311-kube-api-access-x5xqv\") pod \"payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb\" (UID: \"f4dc3ccd-f2c5-40f1-b834-f2a064032311\") " pod="openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb" Nov 03 02:13:42.475761 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:42.475619 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc3ccd-f2c5-40f1-b834-f2a064032311-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb\" (UID: \"f4dc3ccd-f2c5-40f1-b834-f2a064032311\") " pod="openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb" Nov 03 02:13:42.476542 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:42.476517 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc3ccd-f2c5-40f1-b834-f2a064032311-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb\" (UID: \"f4dc3ccd-f2c5-40f1-b834-f2a064032311\") " pod="openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb" Nov 03 02:13:42.532780 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:42.532750 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5xqv\" (UniqueName: \"kubernetes.io/projected/f4dc3ccd-f2c5-40f1-b834-f2a064032311-kube-api-access-x5xqv\") pod \"payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb\" (UID: \"f4dc3ccd-f2c5-40f1-b834-f2a064032311\") " pod="openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb" Nov 03 02:13:42.604046 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:42.604008 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb" Nov 03 02:13:43.001691 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:42.988286 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb"] Nov 03 02:13:43.206961 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:13:43.206317 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:13:43.209152 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:43.209122 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb" event={"ID":"f4dc3ccd-f2c5-40f1-b834-f2a064032311","Type":"ContainerStarted","Data":"c40f43ff668820d334b38a92cb5bc96521963d8439ea43f594216d7cde8cda2c"} Nov 03 02:13:43.427879 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:43.427846 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:44.320251 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:44.320213 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb" event={"ID":"f4dc3ccd-f2c5-40f1-b834-f2a064032311","Type":"ContainerStarted","Data":"e4e9d7e0bfc6f19f89fd4fc92485ffbb3bd436368220c27092a6524b0d812aff"} Nov 03 02:13:44.321833 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:44.321809 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb" Nov 03 02:13:44.465105 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:44.463248 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb" podStartSLOduration=2.463228264 podStartE2EDuration="2.463228264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:13:44.454493455 +0000 UTC m=+1306.138602610" watchObservedRunningTime="2026-11-03 02:13:44.463228264 +0000 UTC m=+1306.147337419" Nov 03 02:13:50.194646 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:50.194609 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:53.184173 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:53.184132 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:56.390372 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:56.390339 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb" Nov 03 02:13:57.823516 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:57.823481 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:59.380831 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:13:59.380788 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:02.535507 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:02.535476 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:07.233002 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:07.232958 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:10.724154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:10.724122 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:12.023661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:12.023626 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:13.007272 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:14:13.007240 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:14:16.682355 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:16.682315 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:16.846925 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:16.846851 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:17.005638 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:17.005555 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:21.171414 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:21.171361 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:21.930332 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:21.930301 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:22.088262 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:22.088222 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:22.566851 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:22.566795 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:22.949148 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:22.949106 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:26.737802 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:26.737768 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:27.111029 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:27.110947 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:27.287000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:27.286965 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:27.447736 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:27.447679 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:27.695586 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:27.695554 5463 patch_prober.go:29] interesting pod/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 02:14:27.695764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:27.695615 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" podUID="9ec04276-3f4d-4bdf-96bf-37f93fd0913e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 02:14:27.829708 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:27.829613 5463 generic.go:356] "Generic (PLEG): container finished" podID="9ec04276-3f4d-4bdf-96bf-37f93fd0913e" containerID="bdc92936e7c5552b0cdaa645cabf9cb7a0f50f330262a1963352820dfa82ae06" exitCode=0 Nov 03 02:14:27.829708 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:27.829684 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" event={"ID":"9ec04276-3f4d-4bdf-96bf-37f93fd0913e","Type":"ContainerDied","Data":"bdc92936e7c5552b0cdaa645cabf9cb7a0f50f330262a1963352820dfa82ae06"} Nov 03 02:14:27.830257 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:27.830110 5463 scope.go:122] "RemoveContainer" containerID="bdc92936e7c5552b0cdaa645cabf9cb7a0f50f330262a1963352820dfa82ae06" Nov 03 02:14:27.831864 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:27.831827 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:14:27.831986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:27.831916 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:14:27.831986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:27.831960 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:14:28.839231 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:28.839168 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" event={"ID":"9ec04276-3f4d-4bdf-96bf-37f93fd0913e","Type":"ContainerStarted","Data":"8649bcda78befe6cb20fc947e18bfc41f77ed6a468eb1bd07e501eb688529f8d"} Nov 03 02:14:29.272558 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:29.272434 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:14:29.272788 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:29.272738 5463 patch_prober.go:29] interesting pod/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 02:14:29.272868 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:29.272811 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" podUID="9ec04276-3f4d-4bdf-96bf-37f93fd0913e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 02:14:30.277212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:30.276616 5463 patch_prober.go:29] interesting pod/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 02:14:30.277212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:30.276684 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" podUID="9ec04276-3f4d-4bdf-96bf-37f93fd0913e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 02:14:30.371048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:30.371011 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:30.536378 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:30.536275 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:30.800480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:30.800391 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f"] Nov 03 02:14:30.828186 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:30.828157 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f" Nov 03 02:14:30.828450 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:30.828427 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f"] Nov 03 02:14:30.942334 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:30.942297 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7j8l\" (UniqueName: \"kubernetes.io/projected/557af6d4-ae3b-4b48-ae87-75095fe70ec2-kube-api-access-q7j8l\") pod \"payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f\" (UID: \"557af6d4-ae3b-4b48-ae87-75095fe70ec2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f" Nov 03 02:14:30.942534 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:30.942389 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/557af6d4-ae3b-4b48-ae87-75095fe70ec2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f\" (UID: \"557af6d4-ae3b-4b48-ae87-75095fe70ec2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f" Nov 03 02:14:30.999223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:30.967095 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c617de\"" Nov 03 02:14:31.012212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:31.000005 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c617de-dockercfg-vgv4n\"" Nov 03 02:14:31.153222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:31.085387 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/557af6d4-ae3b-4b48-ae87-75095fe70ec2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f\" (UID: \"557af6d4-ae3b-4b48-ae87-75095fe70ec2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f" Nov 03 02:14:31.153222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:31.085716 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7j8l\" (UniqueName: \"kubernetes.io/projected/557af6d4-ae3b-4b48-ae87-75095fe70ec2-kube-api-access-q7j8l\") pod \"payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f\" (UID: \"557af6d4-ae3b-4b48-ae87-75095fe70ec2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f" Nov 03 02:14:31.217260 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:31.188677 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/557af6d4-ae3b-4b48-ae87-75095fe70ec2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f\" (UID: \"557af6d4-ae3b-4b48-ae87-75095fe70ec2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f" Nov 03 02:14:31.285327 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:31.284714 5463 patch_prober.go:29] interesting pod/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 02:14:31.285327 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:31.284777 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" podUID="9ec04276-3f4d-4bdf-96bf-37f93fd0913e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 02:14:31.293273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:31.293241 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd"] Nov 03 02:14:31.296961 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:31.296913 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7j8l\" (UniqueName: \"kubernetes.io/projected/557af6d4-ae3b-4b48-ae87-75095fe70ec2-kube-api-access-q7j8l\") pod \"payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f\" (UID: \"557af6d4-ae3b-4b48-ae87-75095fe70ec2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f" Nov 03 02:14:31.298969 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:31.298950 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd" Nov 03 02:14:31.307923 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:31.307802 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd"] Nov 03 02:14:31.345600 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:31.345545 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5472d731-9bca-404c-a5a4-97370f5b04f7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd\" (UID: \"5472d731-9bca-404c-a5a4-97370f5b04f7\") " pod="openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd" Nov 03 02:14:31.345732 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:31.345717 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56qm6\" (UniqueName: \"kubernetes.io/projected/5472d731-9bca-404c-a5a4-97370f5b04f7-kube-api-access-56qm6\") pod \"payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd\" (UID: \"5472d731-9bca-404c-a5a4-97370f5b04f7\") " pod="openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd" Nov 03 02:14:31.468217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:31.447231 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56qm6\" (UniqueName: \"kubernetes.io/projected/5472d731-9bca-404c-a5a4-97370f5b04f7-kube-api-access-56qm6\") pod \"payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd\" (UID: \"5472d731-9bca-404c-a5a4-97370f5b04f7\") " pod="openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd" Nov 03 02:14:31.468706 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:31.468678 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5472d731-9bca-404c-a5a4-97370f5b04f7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd\" (UID: \"5472d731-9bca-404c-a5a4-97370f5b04f7\") " pod="openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd" Nov 03 02:14:31.469785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:31.469758 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5472d731-9bca-404c-a5a4-97370f5b04f7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd\" (UID: \"5472d731-9bca-404c-a5a4-97370f5b04f7\") " pod="openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd" Nov 03 02:14:31.504314 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:31.504293 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f" Nov 03 02:14:31.525255 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:31.525225 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56qm6\" (UniqueName: \"kubernetes.io/projected/5472d731-9bca-404c-a5a4-97370f5b04f7-kube-api-access-56qm6\") pod \"payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd\" (UID: \"5472d731-9bca-404c-a5a4-97370f5b04f7\") " pod="openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd" Nov 03 02:14:31.618262 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:31.618230 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd" Nov 03 02:14:31.827700 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:31.827668 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:31.945168 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:31.945122 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f"] Nov 03 02:14:32.279264 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:32.278663 5463 patch_prober.go:29] interesting pod/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 02:14:32.279264 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:32.278744 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" podUID="9ec04276-3f4d-4bdf-96bf-37f93fd0913e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 02:14:32.316454 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:14:32.316416 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5472d731_9bca_404c_a5a4_97370f5b04f7.slice/crio-5509b534264cbd6a017257751f91657eee6f0044ed5becd0f677d0ff467008ca WatchSource:0}: Error finding container 5509b534264cbd6a017257751f91657eee6f0044ed5becd0f677d0ff467008ca: Status 404 returned error can't find the container with id 5509b534264cbd6a017257751f91657eee6f0044ed5becd0f677d0ff467008ca Nov 03 02:14:32.320424 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:32.320398 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd"] Nov 03 02:14:32.942234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:32.942131 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd" event={"ID":"5472d731-9bca-404c-a5a4-97370f5b04f7","Type":"ContainerStarted","Data":"6a3566f3b9a32c7cbc95e4991cb48a2dcdd961eb4805358de0414caa001f9274"} Nov 03 02:14:32.942234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:32.942223 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd" event={"ID":"5472d731-9bca-404c-a5a4-97370f5b04f7","Type":"ContainerStarted","Data":"5509b534264cbd6a017257751f91657eee6f0044ed5becd0f677d0ff467008ca"} Nov 03 02:14:32.943768 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:32.943621 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd" Nov 03 02:14:32.953375 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:32.953233 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f" event={"ID":"557af6d4-ae3b-4b48-ae87-75095fe70ec2","Type":"ContainerStarted","Data":"0e58fda70f3435c6a411bdfc3efc41add67e150892052ba85a5c1de17af938cb"} Nov 03 02:14:32.953375 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:32.953304 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f" Nov 03 02:14:32.953375 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:32.953331 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f" event={"ID":"557af6d4-ae3b-4b48-ae87-75095fe70ec2","Type":"ContainerStarted","Data":"202598120f88a67c32aab02bd5a53e092447496705b79fac27f74f3f399e28df"} Nov 03 02:14:33.074212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:33.072119 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd" podStartSLOduration=3.07210058 podStartE2EDuration="3.07210058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:14:33.064571845 +0000 UTC m=+1354.748681002" watchObservedRunningTime="2026-11-03 02:14:33.07210058 +0000 UTC m=+1354.756209735" Nov 03 02:14:33.273268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:33.272577 5463 patch_prober.go:29] interesting pod/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 02:14:33.273268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:33.272641 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" podUID="9ec04276-3f4d-4bdf-96bf-37f93fd0913e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 02:14:34.277157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:34.277128 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:14:34.277651 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:34.277623 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:14:34.278516 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:34.278496 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:14:34.498146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:34.498084 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f" podStartSLOduration=4.498067521 podStartE2EDuration="4.498067521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:14:33.192925717 +0000 UTC m=+1354.877034872" watchObservedRunningTime="2026-11-03 02:14:34.498067521 +0000 UTC m=+1356.182176675" Nov 03 02:14:35.980405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:35.980367 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46"] Nov 03 02:14:35.984317 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:35.984293 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.078137 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.078099 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46"] Nov 03 02:14:36.101636 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.101606 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.101839 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.101749 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr2d9\" (UniqueName: \"kubernetes.io/projected/5673c3cf-3606-42b2-968b-8ccec7e0233a-kube-api-access-gr2d9\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.101909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.101843 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.101909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.101896 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.103111 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.103084 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.103383 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.103361 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.103515 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.103502 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.103763 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.103739 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5673c3cf-3606-42b2-968b-8ccec7e0233a-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.103857 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.103793 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.205157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.205116 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr2d9\" (UniqueName: \"kubernetes.io/projected/5673c3cf-3606-42b2-968b-8ccec7e0233a-kube-api-access-gr2d9\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.205383 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.205182 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.205383 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.205249 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.205383 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.205286 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.205383 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.205323 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.205383 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.205350 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.205669 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.205449 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5673c3cf-3606-42b2-968b-8ccec7e0233a-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.205669 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.205508 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.205669 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.205569 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.205826 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.205702 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.205826 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.205800 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.205950 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.205924 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.206010 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.205996 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.206117 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.206098 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5673c3cf-3606-42b2-968b-8ccec7e0233a-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.207888 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.207870 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.207980 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.207964 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.262133 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.262056 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.262366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.262338 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr2d9\" (UniqueName: \"kubernetes.io/projected/5673c3cf-3606-42b2-968b-8ccec7e0233a-kube-api-access-gr2d9\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.302791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.302762 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:36.527015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.526915 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:36.747604 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:14:36.747496 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5673c3cf_3606_42b2_968b_8ccec7e0233a.slice/crio-185f0a1ee841262816480944d1105454a19d62b4830f0e5bec223b34312b95be WatchSource:0}: Error finding container 185f0a1ee841262816480944d1105454a19d62b4830f0e5bec223b34312b95be: Status 404 returned error can't find the container with id 185f0a1ee841262816480944d1105454a19d62b4830f0e5bec223b34312b95be Nov 03 02:14:36.762215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.750344 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46"] Nov 03 02:14:36.762215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.750776 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:14:36.762215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.750856 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:14:36.762215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:36.750902 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:14:37.010297 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:37.010260 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" event={"ID":"5673c3cf-3606-42b2-968b-8ccec7e0233a","Type":"ContainerStarted","Data":"34bc20f65d467ba0d8ac2f5ca61e960260362053807362858bfbeffb8f54c51f"} Nov 03 02:14:37.010791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:37.010327 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" event={"ID":"5673c3cf-3606-42b2-968b-8ccec7e0233a","Type":"ContainerStarted","Data":"185f0a1ee841262816480944d1105454a19d62b4830f0e5bec223b34312b95be"} Nov 03 02:14:37.130347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:37.130249 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" podStartSLOduration=2.13023522 podStartE2EDuration="2.13023522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:14:37.127243349 +0000 UTC m=+1358.811352504" watchObservedRunningTime="2026-11-03 02:14:37.13023522 +0000 UTC m=+1358.814344368" Nov 03 02:14:37.302986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:37.302944 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:37.304752 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:37.304700 5463 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 02:14:37.304911 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:37.304759 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" podUID="5673c3cf-3606-42b2-968b-8ccec7e0233a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 02:14:38.309216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:38.308923 5463 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 02:14:38.309216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:38.308992 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" podUID="5673c3cf-3606-42b2-968b-8ccec7e0233a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 02:14:39.310308 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:39.309571 5463 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 02:14:39.310308 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:39.309634 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" podUID="5673c3cf-3606-42b2-968b-8ccec7e0233a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 02:14:40.305764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:40.305349 5463 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 02:14:40.305764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:40.305414 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" podUID="5673c3cf-3606-42b2-968b-8ccec7e0233a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 02:14:41.306634 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:41.306596 5463 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 02:14:41.307250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:41.307216 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" podUID="5673c3cf-3606-42b2-968b-8ccec7e0233a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 02:14:41.391215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:41.385419 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:41.579273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:41.577663 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:42.321248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:42.304520 5463 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 02:14:42.321248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:42.304586 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" podUID="5673c3cf-3606-42b2-968b-8ccec7e0233a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 02:14:43.042220 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:14:43.042171 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:14:43.303312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:43.303212 5463 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 02:14:43.303312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:43.303280 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" podUID="5673c3cf-3606-42b2-968b-8ccec7e0233a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 02:14:43.966777 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:43.966744 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd" Nov 03 02:14:43.967303 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:43.966822 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f" Nov 03 02:14:44.304430 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:44.304077 5463 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 02:14:44.304430 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:44.304142 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" podUID="5673c3cf-3606-42b2-968b-8ccec7e0233a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 02:14:45.309612 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:45.309577 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:45.310056 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:45.309974 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:45.310836 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:45.310815 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:14:46.133807 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:46.133755 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:48.251737 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:48.251704 5463 generic.go:356] "Generic (PLEG): container finished" podID="92398fe7-87d2-4c4f-8108-31a05b23c521" containerID="346e364d7f86c01a662ed57bcf73243a146e9f259580707498be7c99a366ee8a" exitCode=0 Nov 03 02:14:48.252176 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:48.251789 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" event={"ID":"92398fe7-87d2-4c4f-8108-31a05b23c521","Type":"ContainerDied","Data":"346e364d7f86c01a662ed57bcf73243a146e9f259580707498be7c99a366ee8a"} Nov 03 02:14:48.252320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:48.252302 5463 scope.go:122] "RemoveContainer" containerID="346e364d7f86c01a662ed57bcf73243a146e9f259580707498be7c99a366ee8a" Nov 03 02:14:48.253888 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:48.253858 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:14:48.253968 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:48.253926 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:14:48.253968 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:48.253959 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:14:49.277901 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:49.277869 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" event={"ID":"92398fe7-87d2-4c4f-8108-31a05b23c521","Type":"ContainerStarted","Data":"9e85bf21ced6c467cbfe0f593b74a7b2e15919e62dcb61fa453855f6b49aaa0b"} Nov 03 02:14:50.223873 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:50.223840 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:14:50.225308 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:50.225285 5463 patch_prober.go:29] interesting pod/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 02:14:50.225415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:50.225346 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" podUID="92398fe7-87d2-4c4f-8108-31a05b23c521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 02:14:50.835695 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:50.835663 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:51.076441 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:51.076409 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:14:51.224168 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:51.224132 5463 patch_prober.go:29] interesting pod/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 02:14:51.224381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:51.224219 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" podUID="92398fe7-87d2-4c4f-8108-31a05b23c521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 02:14:52.223519 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:52.223484 5463 patch_prober.go:29] interesting pod/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 02:14:52.223954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:52.223548 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" podUID="92398fe7-87d2-4c4f-8108-31a05b23c521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 02:14:52.424794 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:52.424758 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:52.738308 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:52.738260 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:53.223495 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:53.223452 5463 patch_prober.go:29] interesting pod/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 02:14:53.223678 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:53.223584 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" podUID="92398fe7-87d2-4c4f-8108-31a05b23c521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 02:14:54.223737 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:54.223702 5463 patch_prober.go:29] interesting pod/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 02:14:54.224130 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:54.223765 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" podUID="92398fe7-87d2-4c4f-8108-31a05b23c521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 02:14:55.228754 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:55.228724 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:14:55.229977 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:55.229957 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:14:55.631407 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:55.631302 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:57.052449 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:57.052402 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk"] Nov 03 02:14:57.100943 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:57.100900 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26fcf768-sm7pk"] Nov 03 02:14:58.445677 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:14:58.445643 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7d23af3-99d2-4694-ae47-aff7d32796ff" path="/var/lib/kubelet/pods/b7d23af3-99d2-4694-ae47-aff7d32796ff/volumes" Nov 03 02:15:00.166384 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.166344 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8"] Nov 03 02:15:00.171533 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.171509 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" Nov 03 02:15:00.172284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.172261 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw"] Nov 03 02:15:00.175421 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.175404 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw" Nov 03 02:15:00.182075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.182050 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt"] Nov 03 02:15:00.185300 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.185283 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" Nov 03 02:15:00.216011 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.215988 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz"] Nov 03 02:15:00.218911 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.218895 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz" Nov 03 02:15:00.224287 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.224264 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:15:00.224382 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.224329 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:15:00.227942 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.227920 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:15:00.229610 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.229582 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntm85\" (UniqueName: \"kubernetes.io/projected/77479f36-3f20-4e7c-9406-0332b1b89ae9-kube-api-access-ntm85\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8\" (UID: \"77479f36-3f20-4e7c-9406-0332b1b89ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" Nov 03 02:15:00.229712 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.229692 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bslt\" (UniqueName: \"kubernetes.io/projected/fc2f30fc-b6f8-4218-9516-baf7f4f93015-kube-api-access-2bslt\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw\" (UID: \"fc2f30fc-b6f8-4218-9516-baf7f4f93015\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw" Nov 03 02:15:00.229842 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.229822 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77479f36-3f20-4e7c-9406-0332b1b89ae9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8\" (UID: \"77479f36-3f20-4e7c-9406-0332b1b89ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" Nov 03 02:15:00.229884 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.229857 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc2f30fc-b6f8-4218-9516-baf7f4f93015-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw\" (UID: \"fc2f30fc-b6f8-4218-9516-baf7f4f93015\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw" Nov 03 02:15:00.270593 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.270553 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw"] Nov 03 02:15:00.270881 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.270856 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8"] Nov 03 02:15:00.309434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.309380 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt"] Nov 03 02:15:00.315536 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.315502 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz"] Nov 03 02:15:00.331555 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.331526 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntm85\" (UniqueName: \"kubernetes.io/projected/77479f36-3f20-4e7c-9406-0332b1b89ae9-kube-api-access-ntm85\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8\" (UID: \"77479f36-3f20-4e7c-9406-0332b1b89ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" Nov 03 02:15:00.331705 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.331577 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bslt\" (UniqueName: \"kubernetes.io/projected/fc2f30fc-b6f8-4218-9516-baf7f4f93015-kube-api-access-2bslt\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw\" (UID: \"fc2f30fc-b6f8-4218-9516-baf7f4f93015\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw" Nov 03 02:15:00.331705 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.331630 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb47517-8ca7-4a72-b1a0-48d5d628bc36-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894535-j7wlt\" (UID: \"fdb47517-8ca7-4a72-b1a0-48d5d628bc36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" Nov 03 02:15:00.331705 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.331669 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77479f36-3f20-4e7c-9406-0332b1b89ae9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8\" (UID: \"77479f36-3f20-4e7c-9406-0332b1b89ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" Nov 03 02:15:00.331869 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.331724 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc2f30fc-b6f8-4218-9516-baf7f4f93015-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw\" (UID: \"fc2f30fc-b6f8-4218-9516-baf7f4f93015\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw" Nov 03 02:15:00.331869 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.331778 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vws7l\" (UniqueName: \"kubernetes.io/projected/fdb47517-8ca7-4a72-b1a0-48d5d628bc36-kube-api-access-vws7l\") pod \"maas-api-key-cleanup-29894535-j7wlt\" (UID: \"fdb47517-8ca7-4a72-b1a0-48d5d628bc36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" Nov 03 02:15:00.331869 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.331829 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef65b78-5fba-49c2-a1f2-e8997437770b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz\" (UID: \"bef65b78-5fba-49c2-a1f2-e8997437770b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz" Nov 03 02:15:00.332011 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.331894 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj62j\" (UniqueName: \"kubernetes.io/projected/bef65b78-5fba-49c2-a1f2-e8997437770b-kube-api-access-cj62j\") pod \"maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz\" (UID: \"bef65b78-5fba-49c2-a1f2-e8997437770b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz" Nov 03 02:15:00.332455 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.332423 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77479f36-3f20-4e7c-9406-0332b1b89ae9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8\" (UID: \"77479f36-3f20-4e7c-9406-0332b1b89ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" Nov 03 02:15:00.332546 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.332455 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc2f30fc-b6f8-4218-9516-baf7f4f93015-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw\" (UID: \"fc2f30fc-b6f8-4218-9516-baf7f4f93015\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw" Nov 03 02:15:00.342612 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.342579 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:00.390398 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.390367 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bslt\" (UniqueName: \"kubernetes.io/projected/fc2f30fc-b6f8-4218-9516-baf7f4f93015-kube-api-access-2bslt\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw\" (UID: \"fc2f30fc-b6f8-4218-9516-baf7f4f93015\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw" Nov 03 02:15:00.400476 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.400153 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntm85\" (UniqueName: \"kubernetes.io/projected/77479f36-3f20-4e7c-9406-0332b1b89ae9-kube-api-access-ntm85\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8\" (UID: \"77479f36-3f20-4e7c-9406-0332b1b89ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" Nov 03 02:15:00.457292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.442381 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb47517-8ca7-4a72-b1a0-48d5d628bc36-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894535-j7wlt\" (UID: \"fdb47517-8ca7-4a72-b1a0-48d5d628bc36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" Nov 03 02:15:00.457292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.442528 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vws7l\" (UniqueName: \"kubernetes.io/projected/fdb47517-8ca7-4a72-b1a0-48d5d628bc36-kube-api-access-vws7l\") pod \"maas-api-key-cleanup-29894535-j7wlt\" (UID: \"fdb47517-8ca7-4a72-b1a0-48d5d628bc36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" Nov 03 02:15:00.457292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.442614 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef65b78-5fba-49c2-a1f2-e8997437770b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz\" (UID: \"bef65b78-5fba-49c2-a1f2-e8997437770b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz" Nov 03 02:15:00.457292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.442682 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj62j\" (UniqueName: \"kubernetes.io/projected/bef65b78-5fba-49c2-a1f2-e8997437770b-kube-api-access-cj62j\") pod \"maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz\" (UID: \"bef65b78-5fba-49c2-a1f2-e8997437770b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz" Nov 03 02:15:00.457292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.443481 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb47517-8ca7-4a72-b1a0-48d5d628bc36-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894535-j7wlt\" (UID: \"fdb47517-8ca7-4a72-b1a0-48d5d628bc36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" Nov 03 02:15:00.457292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.445081 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef65b78-5fba-49c2-a1f2-e8997437770b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz\" (UID: \"bef65b78-5fba-49c2-a1f2-e8997437770b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz" Nov 03 02:15:00.491147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.491110 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" Nov 03 02:15:00.502362 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.502333 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw" Nov 03 02:15:00.505014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.504993 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj62j\" (UniqueName: \"kubernetes.io/projected/bef65b78-5fba-49c2-a1f2-e8997437770b-kube-api-access-cj62j\") pod \"maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz\" (UID: \"bef65b78-5fba-49c2-a1f2-e8997437770b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz" Nov 03 02:15:00.506177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.506154 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vws7l\" (UniqueName: \"kubernetes.io/projected/fdb47517-8ca7-4a72-b1a0-48d5d628bc36-kube-api-access-vws7l\") pod \"maas-api-key-cleanup-29894535-j7wlt\" (UID: \"fdb47517-8ca7-4a72-b1a0-48d5d628bc36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" Nov 03 02:15:00.507638 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.507616 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" Nov 03 02:15:00.531785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.531752 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz" Nov 03 02:15:00.896548 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:15:00.896519 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77479f36_3f20_4e7c_9406_0332b1b89ae9.slice/crio-33f9ed89e7d80cc904e4e7d62b25e3e087c959ee7ea0fe0a660eba179d91d1ba WatchSource:0}: Error finding container 33f9ed89e7d80cc904e4e7d62b25e3e087c959ee7ea0fe0a660eba179d91d1ba: Status 404 returned error can't find the container with id 33f9ed89e7d80cc904e4e7d62b25e3e087c959ee7ea0fe0a660eba179d91d1ba Nov 03 02:15:00.900587 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.900565 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8"] Nov 03 02:15:00.942929 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:15:00.942662 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2f30fc_b6f8_4218_9516_baf7f4f93015.slice/crio-bd57fdf84e205df315ef2d88654d4183619151e546bf40eff932d746bcf98ddb WatchSource:0}: Error finding container bd57fdf84e205df315ef2d88654d4183619151e546bf40eff932d746bcf98ddb: Status 404 returned error can't find the container with id bd57fdf84e205df315ef2d88654d4183619151e546bf40eff932d746bcf98ddb Nov 03 02:15:00.952350 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.950140 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw"] Nov 03 02:15:00.954626 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:15:00.954602 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb47517_8ca7_4a72_b1a0_48d5d628bc36.slice/crio-610134badd172c35ca2042c25235e9317cd8419535edb6cfc4b0ad29457a600a WatchSource:0}: Error finding container 610134badd172c35ca2042c25235e9317cd8419535edb6cfc4b0ad29457a600a: Status 404 returned error can't find the container with id 610134badd172c35ca2042c25235e9317cd8419535edb6cfc4b0ad29457a600a Nov 03 02:15:00.956526 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.956506 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz"] Nov 03 02:15:00.962734 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:00.962695 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt"] Nov 03 02:15:01.052015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:01.051992 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc"] Nov 03 02:15:01.059225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:01.059161 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20177b10-r27pc"] Nov 03 02:15:01.382132 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:01.382097 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" event={"ID":"77479f36-3f20-4e7c-9406-0332b1b89ae9","Type":"ContainerStarted","Data":"08bbc036cdfaf724b0df458c44d5e5df568245f2fd808ea1e882991aa58d6ddb"} Nov 03 02:15:01.382571 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:01.382165 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" event={"ID":"77479f36-3f20-4e7c-9406-0332b1b89ae9","Type":"ContainerStarted","Data":"33f9ed89e7d80cc904e4e7d62b25e3e087c959ee7ea0fe0a660eba179d91d1ba"} Nov 03 02:15:01.383628 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:01.383593 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz" event={"ID":"bef65b78-5fba-49c2-a1f2-e8997437770b","Type":"ContainerStarted","Data":"46805fbc6fa9bc011be7298c02064d1332c4554c3be10998fb944cc23dab26fe"} Nov 03 02:15:01.385251 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:01.385218 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" event={"ID":"fdb47517-8ca7-4a72-b1a0-48d5d628bc36","Type":"ContainerStarted","Data":"fcc556c5a1ac73b586309a7bc70196c6796ee6d3eedda4a887247e1f13a16020"} Nov 03 02:15:01.385417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:01.385276 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" event={"ID":"fdb47517-8ca7-4a72-b1a0-48d5d628bc36","Type":"ContainerStarted","Data":"610134badd172c35ca2042c25235e9317cd8419535edb6cfc4b0ad29457a600a"} Nov 03 02:15:01.386772 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:01.386747 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw" event={"ID":"fc2f30fc-b6f8-4218-9516-baf7f4f93015","Type":"ContainerStarted","Data":"bd57fdf84e205df315ef2d88654d4183619151e546bf40eff932d746bcf98ddb"} Nov 03 02:15:01.497535 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:01.497483 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" podStartSLOduration=1.497468888 podStartE2EDuration="1.497468888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:01.495014817 +0000 UTC m=+1383.179123969" watchObservedRunningTime="2026-11-03 02:15:01.497468888 +0000 UTC m=+1383.181578068" Nov 03 02:15:01.610981 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:01.610879 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" podStartSLOduration=1.6108652559999999 podStartE2EDuration="1.610865256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:01.609789164 +0000 UTC m=+1383.293898317" watchObservedRunningTime="2026-11-03 02:15:01.610865256 +0000 UTC m=+1383.294974405" Nov 03 02:15:02.399831 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:02.399798 5463 generic.go:356] "Generic (PLEG): container finished" podID="77479f36-3f20-4e7c-9406-0332b1b89ae9" containerID="08bbc036cdfaf724b0df458c44d5e5df568245f2fd808ea1e882991aa58d6ddb" exitCode=60 Nov 03 02:15:02.400303 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:02.399882 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" event={"ID":"77479f36-3f20-4e7c-9406-0332b1b89ae9","Type":"ContainerDied","Data":"08bbc036cdfaf724b0df458c44d5e5df568245f2fd808ea1e882991aa58d6ddb"} Nov 03 02:15:02.400303 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:02.400159 5463 scope.go:122] "RemoveContainer" containerID="08bbc036cdfaf724b0df458c44d5e5df568245f2fd808ea1e882991aa58d6ddb" Nov 03 02:15:02.401734 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:02.401707 5463 generic.go:356] "Generic (PLEG): container finished" podID="bef65b78-5fba-49c2-a1f2-e8997437770b" containerID="d8954156ca8e578bbe1b6d252db6a99e45ee77775ef3bc11bcff3487512969d0" exitCode=60 Nov 03 02:15:02.401856 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:02.401791 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz" event={"ID":"bef65b78-5fba-49c2-a1f2-e8997437770b","Type":"ContainerDied","Data":"d8954156ca8e578bbe1b6d252db6a99e45ee77775ef3bc11bcff3487512969d0"} Nov 03 02:15:02.402080 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:02.402058 5463 scope.go:122] "RemoveContainer" containerID="d8954156ca8e578bbe1b6d252db6a99e45ee77775ef3bc11bcff3487512969d0" Nov 03 02:15:02.403434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:02.403372 5463 generic.go:356] "Generic (PLEG): container finished" podID="fdb47517-8ca7-4a72-b1a0-48d5d628bc36" containerID="fcc556c5a1ac73b586309a7bc70196c6796ee6d3eedda4a887247e1f13a16020" exitCode=60 Nov 03 02:15:02.403434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:02.403392 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" event={"ID":"fdb47517-8ca7-4a72-b1a0-48d5d628bc36","Type":"ContainerDied","Data":"fcc556c5a1ac73b586309a7bc70196c6796ee6d3eedda4a887247e1f13a16020"} Nov 03 02:15:02.403711 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:02.403696 5463 scope.go:122] "RemoveContainer" containerID="fcc556c5a1ac73b586309a7bc70196c6796ee6d3eedda4a887247e1f13a16020" Nov 03 02:15:02.405361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:02.405345 5463 generic.go:356] "Generic (PLEG): container finished" podID="fc2f30fc-b6f8-4218-9516-baf7f4f93015" containerID="13badec2edb3a58e43bd2ce0c6e5c9434a5394d10c0f8fd54381b3e98cebaf12" exitCode=60 Nov 03 02:15:02.405466 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:02.405410 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw" event={"ID":"fc2f30fc-b6f8-4218-9516-baf7f4f93015","Type":"ContainerDied","Data":"13badec2edb3a58e43bd2ce0c6e5c9434a5394d10c0f8fd54381b3e98cebaf12"} Nov 03 02:15:02.405595 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:02.405579 5463 scope.go:122] "RemoveContainer" containerID="13badec2edb3a58e43bd2ce0c6e5c9434a5394d10c0f8fd54381b3e98cebaf12" Nov 03 02:15:02.445594 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:02.445569 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5877bb76-8962-42bd-b4e6-5b1ac38978d2" path="/var/lib/kubelet/pods/5877bb76-8962-42bd-b4e6-5b1ac38978d2/volumes" Nov 03 02:15:03.415290 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:03.415241 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" event={"ID":"77479f36-3f20-4e7c-9406-0332b1b89ae9","Type":"ContainerStarted","Data":"1d9491008a8c57997923f0b92026862c25706ab0bb8ceea23f0872bd4d913e40"} Nov 03 02:15:03.417587 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:03.417552 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz" event={"ID":"bef65b78-5fba-49c2-a1f2-e8997437770b","Type":"ContainerStarted","Data":"abf6b3d5c221be9eaa5c867c8362dcf110ccb3df5fe9f0a4f6b5255b2ed73875"} Nov 03 02:15:03.419656 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:03.419628 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" event={"ID":"fdb47517-8ca7-4a72-b1a0-48d5d628bc36","Type":"ContainerStarted","Data":"2a7a8935908dac2a3ddef2c0823426832ef85fbc4e18d3d1361d3682e79e34b3"} Nov 03 02:15:03.421681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:03.421656 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw" event={"ID":"fc2f30fc-b6f8-4218-9516-baf7f4f93015","Type":"ContainerStarted","Data":"3e60e200f9fa7395a38f5c69abae9b7e84e25cd95866178a9cfa86e87d2e1ad4"} Nov 03 02:15:03.661993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:03.661930 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw" podStartSLOduration=3.661915697 podStartE2EDuration="3.661915697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:03.656860627 +0000 UTC m=+1385.340969782" watchObservedRunningTime="2026-11-03 02:15:03.661915697 +0000 UTC m=+1385.346024851" Nov 03 02:15:03.786452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:03.786401 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz" podStartSLOduration=3.786386989 podStartE2EDuration="3.786386989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:03.781860385 +0000 UTC m=+1385.465969541" watchObservedRunningTime="2026-11-03 02:15:03.786386989 +0000 UTC m=+1385.470496144" Nov 03 02:15:04.458213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:04.450742 5463 generic.go:356] "Generic (PLEG): container finished" podID="bef65b78-5fba-49c2-a1f2-e8997437770b" containerID="abf6b3d5c221be9eaa5c867c8362dcf110ccb3df5fe9f0a4f6b5255b2ed73875" exitCode=60 Nov 03 02:15:04.458213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:04.451357 5463 scope.go:122] "RemoveContainer" containerID="abf6b3d5c221be9eaa5c867c8362dcf110ccb3df5fe9f0a4f6b5255b2ed73875" Nov 03 02:15:04.458213 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:15:04.451764 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz_odh-ai-gateway-infra(bef65b78-5fba-49c2-a1f2-e8997437770b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz" podUID="bef65b78-5fba-49c2-a1f2-e8997437770b" Nov 03 02:15:04.489214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:04.484071 5463 generic.go:356] "Generic (PLEG): container finished" podID="fdb47517-8ca7-4a72-b1a0-48d5d628bc36" containerID="2a7a8935908dac2a3ddef2c0823426832ef85fbc4e18d3d1361d3682e79e34b3" exitCode=60 Nov 03 02:15:04.505220 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:04.501749 5463 scope.go:122] "RemoveContainer" containerID="2a7a8935908dac2a3ddef2c0823426832ef85fbc4e18d3d1361d3682e79e34b3" Nov 03 02:15:04.505220 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:15:04.502232 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894535-j7wlt_odh-ai-gateway-infra(fdb47517-8ca7-4a72-b1a0-48d5d628bc36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" podUID="fdb47517-8ca7-4a72-b1a0-48d5d628bc36" Nov 03 02:15:04.509214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:04.507979 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz" event={"ID":"bef65b78-5fba-49c2-a1f2-e8997437770b","Type":"ContainerDied","Data":"abf6b3d5c221be9eaa5c867c8362dcf110ccb3df5fe9f0a4f6b5255b2ed73875"} Nov 03 02:15:04.509214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:04.508061 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" event={"ID":"fdb47517-8ca7-4a72-b1a0-48d5d628bc36","Type":"ContainerDied","Data":"2a7a8935908dac2a3ddef2c0823426832ef85fbc4e18d3d1361d3682e79e34b3"} Nov 03 02:15:04.509214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:04.508168 5463 scope.go:122] "RemoveContainer" containerID="d8954156ca8e578bbe1b6d252db6a99e45ee77775ef3bc11bcff3487512969d0" Nov 03 02:15:04.577230 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:04.575505 5463 generic.go:356] "Generic (PLEG): container finished" podID="fc2f30fc-b6f8-4218-9516-baf7f4f93015" containerID="3e60e200f9fa7395a38f5c69abae9b7e84e25cd95866178a9cfa86e87d2e1ad4" exitCode=60 Nov 03 02:15:04.577230 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:04.575619 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw" event={"ID":"fc2f30fc-b6f8-4218-9516-baf7f4f93015","Type":"ContainerDied","Data":"3e60e200f9fa7395a38f5c69abae9b7e84e25cd95866178a9cfa86e87d2e1ad4"} Nov 03 02:15:04.578929 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:04.578539 5463 generic.go:356] "Generic (PLEG): container finished" podID="77479f36-3f20-4e7c-9406-0332b1b89ae9" containerID="1d9491008a8c57997923f0b92026862c25706ab0bb8ceea23f0872bd4d913e40" exitCode=60 Nov 03 02:15:04.578929 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:04.578697 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" event={"ID":"77479f36-3f20-4e7c-9406-0332b1b89ae9","Type":"ContainerDied","Data":"1d9491008a8c57997923f0b92026862c25706ab0bb8ceea23f0872bd4d913e40"} Nov 03 02:15:04.579666 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:04.579243 5463 scope.go:122] "RemoveContainer" containerID="3e60e200f9fa7395a38f5c69abae9b7e84e25cd95866178a9cfa86e87d2e1ad4" Nov 03 02:15:04.579666 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:04.579415 5463 scope.go:122] "RemoveContainer" containerID="1d9491008a8c57997923f0b92026862c25706ab0bb8ceea23f0872bd4d913e40" Nov 03 02:15:04.581089 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:15:04.579689 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw_odh-ai-gateway-infra(fc2f30fc-b6f8-4218-9516-baf7f4f93015)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw" podUID="fc2f30fc-b6f8-4218-9516-baf7f4f93015" Nov 03 02:15:04.581089 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:15:04.580288 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8_odh-ai-gateway-infra(77479f36-3f20-4e7c-9406-0332b1b89ae9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" podUID="77479f36-3f20-4e7c-9406-0332b1b89ae9" Nov 03 02:15:04.616949 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:04.612710 5463 scope.go:122] "RemoveContainer" containerID="fcc556c5a1ac73b586309a7bc70196c6796ee6d3eedda4a887247e1f13a16020" Nov 03 02:15:04.670721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:04.670700 5463 scope.go:122] "RemoveContainer" containerID="13badec2edb3a58e43bd2ce0c6e5c9434a5394d10c0f8fd54381b3e98cebaf12" Nov 03 02:15:04.686931 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:04.686910 5463 scope.go:122] "RemoveContainer" containerID="08bbc036cdfaf724b0df458c44d5e5df568245f2fd808ea1e882991aa58d6ddb" Nov 03 02:15:05.266227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:05.256601 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:05.590521 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:05.590436 5463 scope.go:122] "RemoveContainer" containerID="1d9491008a8c57997923f0b92026862c25706ab0bb8ceea23f0872bd4d913e40" Nov 03 02:15:05.591008 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:15:05.590819 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8_odh-ai-gateway-infra(77479f36-3f20-4e7c-9406-0332b1b89ae9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" podUID="77479f36-3f20-4e7c-9406-0332b1b89ae9" Nov 03 02:15:05.592487 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:05.592459 5463 scope.go:122] "RemoveContainer" containerID="abf6b3d5c221be9eaa5c867c8362dcf110ccb3df5fe9f0a4f6b5255b2ed73875" Nov 03 02:15:05.592831 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:15:05.592800 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz_odh-ai-gateway-infra(bef65b78-5fba-49c2-a1f2-e8997437770b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz" podUID="bef65b78-5fba-49c2-a1f2-e8997437770b" Nov 03 02:15:05.594318 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:05.594296 5463 scope.go:122] "RemoveContainer" containerID="2a7a8935908dac2a3ddef2c0823426832ef85fbc4e18d3d1361d3682e79e34b3" Nov 03 02:15:05.594607 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:15:05.594561 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894535-j7wlt_odh-ai-gateway-infra(fdb47517-8ca7-4a72-b1a0-48d5d628bc36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" podUID="fdb47517-8ca7-4a72-b1a0-48d5d628bc36" Nov 03 02:15:05.596274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:05.596257 5463 scope.go:122] "RemoveContainer" containerID="3e60e200f9fa7395a38f5c69abae9b7e84e25cd95866178a9cfa86e87d2e1ad4" Nov 03 02:15:05.596506 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:15:05.596488 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw_odh-ai-gateway-infra(fc2f30fc-b6f8-4218-9516-baf7f4f93015)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw" podUID="fc2f30fc-b6f8-4218-9516-baf7f4f93015" Nov 03 02:15:09.872112 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:09.872073 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:10.276208 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:10.276156 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:10.438133 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:10.438053 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:10.592923 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:10.592826 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:10.749495 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:10.749463 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:11.072423 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:11.072381 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:11.385821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:11.383570 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:11.665873 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:11.665839 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:12.048445 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:12.048359 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d"] Nov 03 02:15:12.098367 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:12.098336 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dfbda7e-7h26d"] Nov 03 02:15:12.446085 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:12.446051 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="661ba085-9a33-4617-a663-5deba830d1fd" path="/var/lib/kubelet/pods/661ba085-9a33-4617-a663-5deba830d1fd/volumes" Nov 03 02:15:13.009885 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:15:13.009848 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:15:14.003596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:14.003561 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:15.436044 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:15.436010 5463 scope.go:122] "RemoveContainer" containerID="abf6b3d5c221be9eaa5c867c8362dcf110ccb3df5fe9f0a4f6b5255b2ed73875" Nov 03 02:15:15.436525 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:15.436127 5463 scope.go:122] "RemoveContainer" containerID="1d9491008a8c57997923f0b92026862c25706ab0bb8ceea23f0872bd4d913e40" Nov 03 02:15:15.436525 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:15.436267 5463 scope.go:122] "RemoveContainer" containerID="3e60e200f9fa7395a38f5c69abae9b7e84e25cd95866178a9cfa86e87d2e1ad4" Nov 03 02:15:15.436525 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:15.436515 5463 scope.go:122] "RemoveContainer" containerID="2a7a8935908dac2a3ddef2c0823426832ef85fbc4e18d3d1361d3682e79e34b3" Nov 03 02:15:15.518391 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:15.518357 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:15.670808 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:15.670770 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:15.852123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:15.852082 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:16.017619 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:16.017555 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:16.695743 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:16.695714 5463 generic.go:356] "Generic (PLEG): container finished" podID="77479f36-3f20-4e7c-9406-0332b1b89ae9" containerID="5d0f27ad56835924eee45e8df993ce61224c715bec0987fa5e84fea911fcbc8a" exitCode=60 Nov 03 02:15:16.696121 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:16.695791 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" event={"ID":"77479f36-3f20-4e7c-9406-0332b1b89ae9","Type":"ContainerDied","Data":"5d0f27ad56835924eee45e8df993ce61224c715bec0987fa5e84fea911fcbc8a"} Nov 03 02:15:16.696121 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:16.695847 5463 scope.go:122] "RemoveContainer" containerID="1d9491008a8c57997923f0b92026862c25706ab0bb8ceea23f0872bd4d913e40" Nov 03 02:15:16.696275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:16.696259 5463 scope.go:122] "RemoveContainer" containerID="5d0f27ad56835924eee45e8df993ce61224c715bec0987fa5e84fea911fcbc8a" Nov 03 02:15:16.696611 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:15:16.696582 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8_odh-ai-gateway-infra(77479f36-3f20-4e7c-9406-0332b1b89ae9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" podUID="77479f36-3f20-4e7c-9406-0332b1b89ae9" Nov 03 02:15:16.697987 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:16.697961 5463 generic.go:356] "Generic (PLEG): container finished" podID="bef65b78-5fba-49c2-a1f2-e8997437770b" containerID="b79b9b335780ef265fe25c7379a2399fa9840136ae6b224222ecaa944a6d8124" exitCode=60 Nov 03 02:15:16.698067 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:16.698027 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz" event={"ID":"bef65b78-5fba-49c2-a1f2-e8997437770b","Type":"ContainerDied","Data":"b79b9b335780ef265fe25c7379a2399fa9840136ae6b224222ecaa944a6d8124"} Nov 03 02:15:16.698384 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:16.698369 5463 scope.go:122] "RemoveContainer" containerID="b79b9b335780ef265fe25c7379a2399fa9840136ae6b224222ecaa944a6d8124" Nov 03 02:15:16.698699 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:15:16.698681 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz_odh-ai-gateway-infra(bef65b78-5fba-49c2-a1f2-e8997437770b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz" podUID="bef65b78-5fba-49c2-a1f2-e8997437770b" Nov 03 02:15:16.699982 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:16.699963 5463 generic.go:356] "Generic (PLEG): container finished" podID="fdb47517-8ca7-4a72-b1a0-48d5d628bc36" containerID="f1854da4f4a1c3a2008ee2e06820fb1bc7c2b17689a439eac4e827b23b26c883" exitCode=60 Nov 03 02:15:16.700084 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:16.700027 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" event={"ID":"fdb47517-8ca7-4a72-b1a0-48d5d628bc36","Type":"ContainerDied","Data":"f1854da4f4a1c3a2008ee2e06820fb1bc7c2b17689a439eac4e827b23b26c883"} Nov 03 02:15:16.700337 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:16.700321 5463 scope.go:122] "RemoveContainer" containerID="f1854da4f4a1c3a2008ee2e06820fb1bc7c2b17689a439eac4e827b23b26c883" Nov 03 02:15:16.700587 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:15:16.700562 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894535-j7wlt_odh-ai-gateway-infra(fdb47517-8ca7-4a72-b1a0-48d5d628bc36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" podUID="fdb47517-8ca7-4a72-b1a0-48d5d628bc36" Nov 03 02:15:16.702073 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:16.702051 5463 generic.go:356] "Generic (PLEG): container finished" podID="fc2f30fc-b6f8-4218-9516-baf7f4f93015" containerID="f81c39e4ff77eabb9ec48d07607e95db82f178bfbca4e1e7a1e60e65b4198e84" exitCode=60 Nov 03 02:15:16.702141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:16.702103 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw" event={"ID":"fc2f30fc-b6f8-4218-9516-baf7f4f93015","Type":"ContainerDied","Data":"f81c39e4ff77eabb9ec48d07607e95db82f178bfbca4e1e7a1e60e65b4198e84"} Nov 03 02:15:16.702396 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:16.702382 5463 scope.go:122] "RemoveContainer" containerID="f81c39e4ff77eabb9ec48d07607e95db82f178bfbca4e1e7a1e60e65b4198e84" Nov 03 02:15:16.702601 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:15:16.702586 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw_odh-ai-gateway-infra(fc2f30fc-b6f8-4218-9516-baf7f4f93015)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw" podUID="fc2f30fc-b6f8-4218-9516-baf7f4f93015" Nov 03 02:15:16.709859 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:16.709840 5463 scope.go:122] "RemoveContainer" containerID="abf6b3d5c221be9eaa5c867c8362dcf110ccb3df5fe9f0a4f6b5255b2ed73875" Nov 03 02:15:16.724769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:16.724755 5463 scope.go:122] "RemoveContainer" containerID="2a7a8935908dac2a3ddef2c0823426832ef85fbc4e18d3d1361d3682e79e34b3" Nov 03 02:15:16.737397 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:16.737380 5463 scope.go:122] "RemoveContainer" containerID="3e60e200f9fa7395a38f5c69abae9b7e84e25cd95866178a9cfa86e87d2e1ad4" Nov 03 02:15:17.831469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:17.831438 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8"] Nov 03 02:15:17.945899 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:17.945863 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz"] Nov 03 02:15:18.057243 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.056367 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" Nov 03 02:15:18.073098 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.060889 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt"] Nov 03 02:15:18.101474 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.101218 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77479f36-3f20-4e7c-9406-0332b1b89ae9-kube-api-access-ntm85\" (UniqueName: \"kubernetes.io/projected/77479f36-3f20-4e7c-9406-0332b1b89ae9-kube-api-access-ntm85\") pod \"77479f36-3f20-4e7c-9406-0332b1b89ae9\" (UID: \"77479f36-3f20-4e7c-9406-0332b1b89ae9\") " Nov 03 02:15:18.101474 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.101326 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77479f36-3f20-4e7c-9406-0332b1b89ae9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77479f36-3f20-4e7c-9406-0332b1b89ae9-maas-api-service-ca\") pod \"77479f36-3f20-4e7c-9406-0332b1b89ae9\" (UID: \"77479f36-3f20-4e7c-9406-0332b1b89ae9\") " Nov 03 02:15:18.101788 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.101752 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77479f36-3f20-4e7c-9406-0332b1b89ae9-maas-api-service-ca" pod "77479f36-3f20-4e7c-9406-0332b1b89ae9" (UID: "77479f36-3f20-4e7c-9406-0332b1b89ae9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:18.102221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.102184 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77479f36-3f20-4e7c-9406-0332b1b89ae9-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:18.124924 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.124484 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77479f36-3f20-4e7c-9406-0332b1b89ae9-kube-api-access-ntm85" pod "77479f36-3f20-4e7c-9406-0332b1b89ae9" (UID: "77479f36-3f20-4e7c-9406-0332b1b89ae9"). InnerVolumeSpecName "kube-api-access-ntm85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:18.202579 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.201766 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw"] Nov 03 02:15:18.204433 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.204396 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntm85\" (UniqueName: \"kubernetes.io/projected/77479f36-3f20-4e7c-9406-0332b1b89ae9-kube-api-access-ntm85\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:18.217008 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.211808 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz" Nov 03 02:15:18.267159 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.264657 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" Nov 03 02:15:18.305701 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.305669 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bef65b78-5fba-49c2-a1f2-e8997437770b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef65b78-5fba-49c2-a1f2-e8997437770b-maas-api-service-ca\") pod \"bef65b78-5fba-49c2-a1f2-e8997437770b\" (UID: \"bef65b78-5fba-49c2-a1f2-e8997437770b\") " Nov 03 02:15:18.305896 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.305873 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef65b78-5fba-49c2-a1f2-e8997437770b-kube-api-access-cj62j\" (UniqueName: \"kubernetes.io/projected/bef65b78-5fba-49c2-a1f2-e8997437770b-kube-api-access-cj62j\") pod \"bef65b78-5fba-49c2-a1f2-e8997437770b\" (UID: \"bef65b78-5fba-49c2-a1f2-e8997437770b\") " Nov 03 02:15:18.306087 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.306064 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef65b78-5fba-49c2-a1f2-e8997437770b-maas-api-service-ca" pod "bef65b78-5fba-49c2-a1f2-e8997437770b" (UID: "bef65b78-5fba-49c2-a1f2-e8997437770b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:18.306716 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.306691 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef65b78-5fba-49c2-a1f2-e8997437770b-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:18.308523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.308477 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef65b78-5fba-49c2-a1f2-e8997437770b-kube-api-access-cj62j" pod "bef65b78-5fba-49c2-a1f2-e8997437770b" (UID: "bef65b78-5fba-49c2-a1f2-e8997437770b"). InnerVolumeSpecName "kube-api-access-cj62j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:18.407441 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.407410 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb47517-8ca7-4a72-b1a0-48d5d628bc36-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb47517-8ca7-4a72-b1a0-48d5d628bc36-maas-api-service-ca\") pod \"fdb47517-8ca7-4a72-b1a0-48d5d628bc36\" (UID: \"fdb47517-8ca7-4a72-b1a0-48d5d628bc36\") " Nov 03 02:15:18.407591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.407459 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb47517-8ca7-4a72-b1a0-48d5d628bc36-kube-api-access-vws7l\" (UniqueName: \"kubernetes.io/projected/fdb47517-8ca7-4a72-b1a0-48d5d628bc36-kube-api-access-vws7l\") pod \"fdb47517-8ca7-4a72-b1a0-48d5d628bc36\" (UID: \"fdb47517-8ca7-4a72-b1a0-48d5d628bc36\") " Nov 03 02:15:18.407755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.407735 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb47517-8ca7-4a72-b1a0-48d5d628bc36-maas-api-service-ca" pod "fdb47517-8ca7-4a72-b1a0-48d5d628bc36" (UID: "fdb47517-8ca7-4a72-b1a0-48d5d628bc36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:18.408219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.408181 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb47517-8ca7-4a72-b1a0-48d5d628bc36-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:18.408304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.408223 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj62j\" (UniqueName: \"kubernetes.io/projected/bef65b78-5fba-49c2-a1f2-e8997437770b-kube-api-access-cj62j\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:18.409128 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.409107 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw" Nov 03 02:15:18.409596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.409576 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb47517-8ca7-4a72-b1a0-48d5d628bc36-kube-api-access-vws7l" pod "fdb47517-8ca7-4a72-b1a0-48d5d628bc36" (UID: "fdb47517-8ca7-4a72-b1a0-48d5d628bc36"). InnerVolumeSpecName "kube-api-access-vws7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:18.508698 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.508671 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc2f30fc-b6f8-4218-9516-baf7f4f93015-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc2f30fc-b6f8-4218-9516-baf7f4f93015-maas-api-service-ca\") pod \"fc2f30fc-b6f8-4218-9516-baf7f4f93015\" (UID: \"fc2f30fc-b6f8-4218-9516-baf7f4f93015\") " Nov 03 02:15:18.508834 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.508795 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc2f30fc-b6f8-4218-9516-baf7f4f93015-kube-api-access-2bslt\" (UniqueName: \"kubernetes.io/projected/fc2f30fc-b6f8-4218-9516-baf7f4f93015-kube-api-access-2bslt\") pod \"fc2f30fc-b6f8-4218-9516-baf7f4f93015\" (UID: \"fc2f30fc-b6f8-4218-9516-baf7f4f93015\") " Nov 03 02:15:18.509060 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.509037 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc2f30fc-b6f8-4218-9516-baf7f4f93015-maas-api-service-ca" pod "fc2f30fc-b6f8-4218-9516-baf7f4f93015" (UID: "fc2f30fc-b6f8-4218-9516-baf7f4f93015"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:18.509426 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.509407 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vws7l\" (UniqueName: \"kubernetes.io/projected/fdb47517-8ca7-4a72-b1a0-48d5d628bc36-kube-api-access-vws7l\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:18.509500 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.509429 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc2f30fc-b6f8-4218-9516-baf7f4f93015-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:18.511009 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.510989 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2f30fc-b6f8-4218-9516-baf7f4f93015-kube-api-access-2bslt" pod "fc2f30fc-b6f8-4218-9516-baf7f4f93015" (UID: "fc2f30fc-b6f8-4218-9516-baf7f4f93015"). InnerVolumeSpecName "kube-api-access-2bslt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:18.610679 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.610641 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bslt\" (UniqueName: \"kubernetes.io/projected/fc2f30fc-b6f8-4218-9516-baf7f4f93015-kube-api-access-2bslt\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:18.726973 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.726896 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" Nov 03 02:15:18.726973 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.726925 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8" event={"ID":"77479f36-3f20-4e7c-9406-0332b1b89ae9","Type":"ContainerDied","Data":"33f9ed89e7d80cc904e4e7d62b25e3e087c959ee7ea0fe0a660eba179d91d1ba"} Nov 03 02:15:18.727208 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.727018 5463 scope.go:122] "RemoveContainer" containerID="5d0f27ad56835924eee45e8df993ce61224c715bec0987fa5e84fea911fcbc8a" Nov 03 02:15:18.728840 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.728649 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz" Nov 03 02:15:18.728840 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.728673 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz" event={"ID":"bef65b78-5fba-49c2-a1f2-e8997437770b","Type":"ContainerDied","Data":"46805fbc6fa9bc011be7298c02064d1332c4554c3be10998fb944cc23dab26fe"} Nov 03 02:15:18.730430 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.730410 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" event={"ID":"fdb47517-8ca7-4a72-b1a0-48d5d628bc36","Type":"ContainerDied","Data":"610134badd172c35ca2042c25235e9317cd8419535edb6cfc4b0ad29457a600a"} Nov 03 02:15:18.730526 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.730483 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt" Nov 03 02:15:18.732243 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.732221 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw" event={"ID":"fc2f30fc-b6f8-4218-9516-baf7f4f93015","Type":"ContainerDied","Data":"bd57fdf84e205df315ef2d88654d4183619151e546bf40eff932d746bcf98ddb"} Nov 03 02:15:18.732367 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.732292 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw" Nov 03 02:15:18.741797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.741779 5463 scope.go:122] "RemoveContainer" containerID="b79b9b335780ef265fe25c7379a2399fa9840136ae6b224222ecaa944a6d8124" Nov 03 02:15:18.751256 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.751237 5463 scope.go:122] "RemoveContainer" containerID="f1854da4f4a1c3a2008ee2e06820fb1bc7c2b17689a439eac4e827b23b26c883" Nov 03 02:15:18.760828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.760812 5463 scope.go:122] "RemoveContainer" containerID="f81c39e4ff77eabb9ec48d07607e95db82f178bfbca4e1e7a1e60e65b4198e84" Nov 03 02:15:18.909702 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.909665 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw"] Nov 03 02:15:18.916057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.916030 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894535-bh6lw"] Nov 03 02:15:18.923982 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:18.923954 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:19.045526 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.035869 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8"] Nov 03 02:15:19.097348 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.096105 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894535-67ms8"] Nov 03 02:15:19.103990 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.103888 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:19.187219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.184448 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz"] Nov 03 02:15:19.193217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.192349 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-701ba30f-29894535-bbjvz"] Nov 03 02:15:19.295749 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.295665 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt"] Nov 03 02:15:19.302606 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.302576 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894535-j7wlt"] Nov 03 02:15:19.325007 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.324977 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488"] Nov 03 02:15:19.326147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326023 5463 cpu_manager.go:414] "containerMap: removing container" podUID="fc2f30fc-b6f8-4218-9516-baf7f4f93015" containerName="cleanup" Nov 03 02:15:19.326147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326047 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc2f30fc-b6f8-4218-9516-baf7f4f93015" containerName="cleanup" Nov 03 02:15:19.326147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326086 5463 cpu_manager.go:414] "containerMap: removing container" podUID="fdb47517-8ca7-4a72-b1a0-48d5d628bc36" containerName="cleanup" Nov 03 02:15:19.326147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326094 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb47517-8ca7-4a72-b1a0-48d5d628bc36" containerName="cleanup" Nov 03 02:15:19.326147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326112 5463 cpu_manager.go:414] "containerMap: removing container" podUID="77479f36-3f20-4e7c-9406-0332b1b89ae9" containerName="cleanup" Nov 03 02:15:19.326147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326120 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77479f36-3f20-4e7c-9406-0332b1b89ae9" containerName="cleanup" Nov 03 02:15:19.326147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326137 5463 cpu_manager.go:414] "containerMap: removing container" podUID="77479f36-3f20-4e7c-9406-0332b1b89ae9" containerName="cleanup" Nov 03 02:15:19.326147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326145 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77479f36-3f20-4e7c-9406-0332b1b89ae9" containerName="cleanup" Nov 03 02:15:19.326664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326173 5463 cpu_manager.go:414] "containerMap: removing container" podUID="fdb47517-8ca7-4a72-b1a0-48d5d628bc36" containerName="cleanup" Nov 03 02:15:19.326664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326180 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb47517-8ca7-4a72-b1a0-48d5d628bc36" containerName="cleanup" Nov 03 02:15:19.326664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326205 5463 cpu_manager.go:414] "containerMap: removing container" podUID="bef65b78-5fba-49c2-a1f2-e8997437770b" containerName="cleanup" Nov 03 02:15:19.326664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326213 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef65b78-5fba-49c2-a1f2-e8997437770b" containerName="cleanup" Nov 03 02:15:19.326664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326233 5463 cpu_manager.go:414] "containerMap: removing container" podUID="bef65b78-5fba-49c2-a1f2-e8997437770b" containerName="cleanup" Nov 03 02:15:19.326664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326240 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef65b78-5fba-49c2-a1f2-e8997437770b" containerName="cleanup" Nov 03 02:15:19.326664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326254 5463 cpu_manager.go:414] "containerMap: removing container" podUID="bef65b78-5fba-49c2-a1f2-e8997437770b" containerName="cleanup" Nov 03 02:15:19.326664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326262 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef65b78-5fba-49c2-a1f2-e8997437770b" containerName="cleanup" Nov 03 02:15:19.326664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326274 5463 cpu_manager.go:414] "containerMap: removing container" podUID="fc2f30fc-b6f8-4218-9516-baf7f4f93015" containerName="cleanup" Nov 03 02:15:19.326664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326281 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc2f30fc-b6f8-4218-9516-baf7f4f93015" containerName="cleanup" Nov 03 02:15:19.326664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326445 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb47517-8ca7-4a72-b1a0-48d5d628bc36" containerName="cleanup" Nov 03 02:15:19.326664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326460 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="77479f36-3f20-4e7c-9406-0332b1b89ae9" containerName="cleanup" Nov 03 02:15:19.326664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326471 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="77479f36-3f20-4e7c-9406-0332b1b89ae9" containerName="cleanup" Nov 03 02:15:19.326664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326486 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb47517-8ca7-4a72-b1a0-48d5d628bc36" containerName="cleanup" Nov 03 02:15:19.326664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326495 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef65b78-5fba-49c2-a1f2-e8997437770b" containerName="cleanup" Nov 03 02:15:19.326664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326505 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc2f30fc-b6f8-4218-9516-baf7f4f93015" containerName="cleanup" Nov 03 02:15:19.326664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326515 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb47517-8ca7-4a72-b1a0-48d5d628bc36" containerName="cleanup" Nov 03 02:15:19.326664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326524 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef65b78-5fba-49c2-a1f2-e8997437770b" containerName="cleanup" Nov 03 02:15:19.326664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.326533 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc2f30fc-b6f8-4218-9516-baf7f4f93015" containerName="cleanup" Nov 03 02:15:19.332445 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.332426 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488" Nov 03 02:15:19.378568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.378542 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488"] Nov 03 02:15:19.388235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.388170 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-aa82d2\"" Nov 03 02:15:19.389491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.389467 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-aa82d2-dockercfg-w4zw5\"" Nov 03 02:15:19.501853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.501810 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krbhb\" (UniqueName: \"kubernetes.io/projected/1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0-kube-api-access-krbhb\") pod \"payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488\" (UID: \"1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488" Nov 03 02:15:19.502333 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.502304 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488\" (UID: \"1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488" Nov 03 02:15:19.558572 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.558474 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6"] Nov 03 02:15:19.563091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.559724 5463 cpu_manager.go:414] "containerMap: removing container" podUID="77479f36-3f20-4e7c-9406-0332b1b89ae9" containerName="cleanup" Nov 03 02:15:19.563091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.559749 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77479f36-3f20-4e7c-9406-0332b1b89ae9" containerName="cleanup" Nov 03 02:15:19.563091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.559790 5463 cpu_manager.go:414] "containerMap: removing container" podUID="fdb47517-8ca7-4a72-b1a0-48d5d628bc36" containerName="cleanup" Nov 03 02:15:19.563091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.559798 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb47517-8ca7-4a72-b1a0-48d5d628bc36" containerName="cleanup" Nov 03 02:15:19.563091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.559830 5463 cpu_manager.go:414] "containerMap: removing container" podUID="fc2f30fc-b6f8-4218-9516-baf7f4f93015" containerName="cleanup" Nov 03 02:15:19.563091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.559838 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc2f30fc-b6f8-4218-9516-baf7f4f93015" containerName="cleanup" Nov 03 02:15:19.563091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.560009 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="77479f36-3f20-4e7c-9406-0332b1b89ae9" containerName="cleanup" Nov 03 02:15:19.563091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.560028 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef65b78-5fba-49c2-a1f2-e8997437770b" containerName="cleanup" Nov 03 02:15:19.563091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.560039 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc2f30fc-b6f8-4218-9516-baf7f4f93015" containerName="cleanup" Nov 03 02:15:19.567673 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.567651 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6"] Nov 03 02:15:19.567784 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.567758 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6" Nov 03 02:15:19.651279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.649822 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krbhb\" (UniqueName: \"kubernetes.io/projected/1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0-kube-api-access-krbhb\") pod \"payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488\" (UID: \"1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488" Nov 03 02:15:19.651279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.649947 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488\" (UID: \"1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488" Nov 03 02:15:19.651279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.650944 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f85293-820a-436c-a860-55b068a4482a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6\" (UID: \"c1f85293-820a-436c-a860-55b068a4482a\") " pod="openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6" Nov 03 02:15:19.651279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.650970 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8gkl\" (UniqueName: \"kubernetes.io/projected/c1f85293-820a-436c-a860-55b068a4482a-kube-api-access-b8gkl\") pod \"payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6\" (UID: \"c1f85293-820a-436c-a860-55b068a4482a\") " pod="openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6" Nov 03 02:15:19.651279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.650862 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488\" (UID: \"1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488" Nov 03 02:15:19.707091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.707062 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krbhb\" (UniqueName: \"kubernetes.io/projected/1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0-kube-api-access-krbhb\") pod \"payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488\" (UID: \"1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488" Nov 03 02:15:19.752348 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.752317 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f85293-820a-436c-a860-55b068a4482a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6\" (UID: \"c1f85293-820a-436c-a860-55b068a4482a\") " pod="openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6" Nov 03 02:15:19.752546 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.752360 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8gkl\" (UniqueName: \"kubernetes.io/projected/c1f85293-820a-436c-a860-55b068a4482a-kube-api-access-b8gkl\") pod \"payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6\" (UID: \"c1f85293-820a-436c-a860-55b068a4482a\") " pod="openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6" Nov 03 02:15:19.753043 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.753018 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f85293-820a-436c-a860-55b068a4482a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6\" (UID: \"c1f85293-820a-436c-a860-55b068a4482a\") " pod="openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6" Nov 03 02:15:19.809576 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.809507 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8gkl\" (UniqueName: \"kubernetes.io/projected/c1f85293-820a-436c-a860-55b068a4482a-kube-api-access-b8gkl\") pod \"payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6\" (UID: \"c1f85293-820a-436c-a860-55b068a4482a\") " pod="openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6" Nov 03 02:15:19.951012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.950981 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488" Nov 03 02:15:19.954375 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:19.954153 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6" Nov 03 02:15:20.340265 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:15:20.340233 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef23d0c_f08f_4bf8_9268_3ea93e6bd7b0.slice/crio-e84e8cf0a6f04df3393f5f47b5686f0d77c037a0bb79894570c63c0cd043f696 WatchSource:0}: Error finding container e84e8cf0a6f04df3393f5f47b5686f0d77c037a0bb79894570c63c0cd043f696: Status 404 returned error can't find the container with id e84e8cf0a6f04df3393f5f47b5686f0d77c037a0bb79894570c63c0cd043f696 Nov 03 02:15:20.348687 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:20.348655 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488"] Nov 03 02:15:20.392117 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:15:20.392085 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f85293_820a_436c_a860_55b068a4482a.slice/crio-f8c64c96dde0a86856516af48e9cb04817035c2ea6b9fe9f8cca6d6e97ddba66 WatchSource:0}: Error finding container f8c64c96dde0a86856516af48e9cb04817035c2ea6b9fe9f8cca6d6e97ddba66: Status 404 returned error can't find the container with id f8c64c96dde0a86856516af48e9cb04817035c2ea6b9fe9f8cca6d6e97ddba66 Nov 03 02:15:20.392310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:20.392246 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6"] Nov 03 02:15:20.444683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:20.444640 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:20.450514 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:20.450490 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77479f36-3f20-4e7c-9406-0332b1b89ae9" path="/var/lib/kubelet/pods/77479f36-3f20-4e7c-9406-0332b1b89ae9/volumes" Nov 03 02:15:20.451200 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:20.451170 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bef65b78-5fba-49c2-a1f2-e8997437770b" path="/var/lib/kubelet/pods/bef65b78-5fba-49c2-a1f2-e8997437770b/volumes" Nov 03 02:15:20.452269 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:20.451805 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc2f30fc-b6f8-4218-9516-baf7f4f93015" path="/var/lib/kubelet/pods/fc2f30fc-b6f8-4218-9516-baf7f4f93015/volumes" Nov 03 02:15:20.452810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:20.452779 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdb47517-8ca7-4a72-b1a0-48d5d628bc36" path="/var/lib/kubelet/pods/fdb47517-8ca7-4a72-b1a0-48d5d628bc36/volumes" Nov 03 02:15:20.586850 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:20.586735 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:20.759297 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:20.759261 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6" event={"ID":"c1f85293-820a-436c-a860-55b068a4482a","Type":"ContainerStarted","Data":"2933f0c0b1b710f237dc9d575f7e7f5a9608488660872177141855185bf04c30"} Nov 03 02:15:20.759514 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:20.759328 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6" event={"ID":"c1f85293-820a-436c-a860-55b068a4482a","Type":"ContainerStarted","Data":"f8c64c96dde0a86856516af48e9cb04817035c2ea6b9fe9f8cca6d6e97ddba66"} Nov 03 02:15:20.759609 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:20.759591 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6" Nov 03 02:15:20.761211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:20.761173 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488" event={"ID":"1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0","Type":"ContainerStarted","Data":"39af84c07019a1095a44c4266e6ec6e5c76eec98d341280077f7edf08fb3b57d"} Nov 03 02:15:20.761356 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:20.761230 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488" event={"ID":"1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0","Type":"ContainerStarted","Data":"e84e8cf0a6f04df3393f5f47b5686f0d77c037a0bb79894570c63c0cd043f696"} Nov 03 02:15:20.761356 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:20.761274 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488" Nov 03 02:15:20.875773 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:20.875676 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6" podStartSLOduration=1.875659895 podStartE2EDuration="1.875659895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:20.873294575 +0000 UTC m=+1402.557403733" watchObservedRunningTime="2026-11-03 02:15:20.875659895 +0000 UTC m=+1402.559769052" Nov 03 02:15:21.009884 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:21.009780 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488" podStartSLOduration=2.009758582 podStartE2EDuration="2.009758582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:21.001462143 +0000 UTC m=+1402.685571297" watchObservedRunningTime="2026-11-03 02:15:21.009758582 +0000 UTC m=+1402.693867741" Nov 03 02:15:21.058952 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:21.054541 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl"] Nov 03 02:15:21.062063 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:21.062022 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d002e070-d5pfl"] Nov 03 02:15:22.445025 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:22.444993 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9a5a567-bf98-411f-95bf-394a62dfab0d" path="/var/lib/kubelet/pods/d9a5a567-bf98-411f-95bf-394a62dfab0d/volumes" Nov 03 02:15:25.235497 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:25.235445 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:26.888957 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:26.888917 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:27.438558 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.438524 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9"] Nov 03 02:15:27.451428 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.451401 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.620533 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.620503 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6198c6-openshift-default-dockercfg-znmdn\"" Nov 03 02:15:27.640309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.635118 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istiod-ca-cert\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.640309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.635204 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-data\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.640309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.635294 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-workload-certs\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.640309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.635326 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-workload-socket\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.640309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.635447 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-podinfo\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.640309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.635476 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-envoy\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.640309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.635510 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-token\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.640309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.635550 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvm25\" (UniqueName: \"kubernetes.io/projected/2fd41e7e-553d-45c9-b25c-d4aa074c899d-kube-api-access-gvm25\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.640309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.635603 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-credential-socket\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.640941 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.640363 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:27.689265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.683254 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9"] Nov 03 02:15:27.827221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.821559 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-data\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.827221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.821639 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-workload-certs\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.827221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.821680 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-workload-socket\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.827221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.821792 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-podinfo\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.827221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.821824 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-envoy\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.827221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.821860 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-token\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.827221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.821898 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvm25\" (UniqueName: \"kubernetes.io/projected/2fd41e7e-553d-45c9-b25c-d4aa074c899d-kube-api-access-gvm25\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.827221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.821958 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-credential-socket\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.827221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.822143 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istiod-ca-cert\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.827221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.822950 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istiod-ca-cert\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.904376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.874753 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-data\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.904376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.875032 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-workload-certs\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.904376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.875319 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-workload-socket\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.914660 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.905881 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-credential-socket\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.914660 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.910647 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-podinfo\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.914660 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.914549 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-envoy\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:27.995260 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:27.995153 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:28.096233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:28.090098 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h"] Nov 03 02:15:28.139223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:28.138811 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-token\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:28.155274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:28.147457 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvm25\" (UniqueName: \"kubernetes.io/projected/2fd41e7e-553d-45c9-b25c-d4aa074c899d-kube-api-access-gvm25\") pod \"e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:28.155514 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:28.150071 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cf2001e-mqw2h"] Nov 03 02:15:28.178222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:28.176805 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:28.451249 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:28.447231 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd008509-2862-4597-bbd3-9321083f030a" path="/var/lib/kubelet/pods/dd008509-2862-4597-bbd3-9321083f030a/volumes" Nov 03 02:15:28.780135 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:28.780085 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9"] Nov 03 02:15:28.810750 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:15:28.810716 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd41e7e_553d_45c9_b25c_d4aa074c899d.slice/crio-8780147d68f9afe90f91ee1421b64fc56f0491b9d60857bc6b139918110600ae WatchSource:0}: Error finding container 8780147d68f9afe90f91ee1421b64fc56f0491b9d60857bc6b139918110600ae: Status 404 returned error can't find the container with id 8780147d68f9afe90f91ee1421b64fc56f0491b9d60857bc6b139918110600ae Nov 03 02:15:28.814317 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:28.814279 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:15:28.814463 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:28.814357 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:15:28.814463 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:28.814409 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:15:29.011681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:29.011643 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" event={"ID":"2fd41e7e-553d-45c9-b25c-d4aa074c899d","Type":"ContainerStarted","Data":"537e46e20db1abb465a30d70c651246ee409315f3744de348bd838ffee9dfb5a"} Nov 03 02:15:29.012106 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:29.011705 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" event={"ID":"2fd41e7e-553d-45c9-b25c-d4aa074c899d","Type":"ContainerStarted","Data":"8780147d68f9afe90f91ee1421b64fc56f0491b9d60857bc6b139918110600ae"} Nov 03 02:15:29.128483 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:29.128391 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" podStartSLOduration=2.128377319 podStartE2EDuration="2.128377319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:29.127842634 +0000 UTC m=+1410.811951826" watchObservedRunningTime="2026-11-03 02:15:29.128377319 +0000 UTC m=+1410.812486474" Nov 03 02:15:29.179369 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:29.179333 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:29.181100 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:29.181064 5463 patch_prober.go:29] interesting pod/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 02:15:29.181246 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:29.181125 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" podUID="2fd41e7e-553d-45c9-b25c-d4aa074c899d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 02:15:30.033497 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:30.033464 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:30.184902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:30.184869 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:31.028710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:31.028686 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:31.029579 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:31.029557 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:15:31.774686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:31.774655 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488" Nov 03 02:15:31.775342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:31.775322 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6" Nov 03 02:15:31.920152 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:31.920119 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:32.153182 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:32.153150 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:32.360081 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:32.360049 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:34.048212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:34.048161 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm"] Nov 03 02:15:34.097986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:34.097947 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ca6ac6-hqgwm"] Nov 03 02:15:34.444489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:34.444459 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1137541f-61df-4c91-a030-d4c6c8e30778" path="/var/lib/kubelet/pods/1137541f-61df-4c91-a030-d4c6c8e30778/volumes" Nov 03 02:15:35.141762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:35.141725 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:36.424261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.424224 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc"] Nov 03 02:15:36.428757 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.428733 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.469385 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.469357 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.469500 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.469394 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.469552 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.469513 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.469593 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.469557 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.469663 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.469642 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.469713 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.469676 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzw97\" (UniqueName: \"kubernetes.io/projected/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-kube-api-access-vzw97\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.469713 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.469708 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.469891 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.469869 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.469940 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.469903 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.571227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.571168 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.571412 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.571240 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.571412 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.571346 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.571576 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.571419 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.571576 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.571472 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.571576 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.571511 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzw97\" (UniqueName: \"kubernetes.io/projected/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-kube-api-access-vzw97\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.571576 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.571563 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.571796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.571604 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.571796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.571691 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.571796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.571719 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.572438 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.572408 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.572560 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.572445 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.572755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.572730 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.572984 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.572963 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.574607 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.574580 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.575482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.575450 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.606234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.606179 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:36.626159 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.626107 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc"] Nov 03 02:15:36.631265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.631240 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.631565 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.631543 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzw97\" (UniqueName: \"kubernetes.io/projected/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-kube-api-access-vzw97\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.745411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.745300 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:36.814704 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:36.814014 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:37.022262 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.021134 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6"] Nov 03 02:15:37.035213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.025998 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6" Nov 03 02:15:37.087877 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.085529 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6198c6\"" Nov 03 02:15:37.087877 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.086316 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6198c6-dockercfg-k2v6x\"" Nov 03 02:15:37.092469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.090347 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1941d760-2cbe-42ba-a119-d1e4006d3c86-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6\" (UID: \"1941d760-2cbe-42ba-a119-d1e4006d3c86\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6" Nov 03 02:15:37.092469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.090610 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4d6g\" (UniqueName: \"kubernetes.io/projected/1941d760-2cbe-42ba-a119-d1e4006d3c86-kube-api-access-l4d6g\") pod \"payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6\" (UID: \"1941d760-2cbe-42ba-a119-d1e4006d3c86\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6" Nov 03 02:15:37.107393 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.107361 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6"] Nov 03 02:15:37.166137 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.166084 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc"] Nov 03 02:15:37.170074 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.170043 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc" Nov 03 02:15:37.192552 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.192526 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98flq\" (UniqueName: \"kubernetes.io/projected/cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1-kube-api-access-98flq\") pod \"payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc\" (UID: \"cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1\") " pod="openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc" Nov 03 02:15:37.192718 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.192566 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1941d760-2cbe-42ba-a119-d1e4006d3c86-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6\" (UID: \"1941d760-2cbe-42ba-a119-d1e4006d3c86\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6" Nov 03 02:15:37.192718 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.192650 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc\" (UID: \"cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1\") " pod="openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc" Nov 03 02:15:37.192854 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.192808 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4d6g\" (UniqueName: \"kubernetes.io/projected/1941d760-2cbe-42ba-a119-d1e4006d3c86-kube-api-access-l4d6g\") pod \"payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6\" (UID: \"1941d760-2cbe-42ba-a119-d1e4006d3c86\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6" Nov 03 02:15:37.193896 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.193869 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1941d760-2cbe-42ba-a119-d1e4006d3c86-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6\" (UID: \"1941d760-2cbe-42ba-a119-d1e4006d3c86\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6" Nov 03 02:15:37.245361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.242550 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc"] Nov 03 02:15:37.361262 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.358648 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98flq\" (UniqueName: \"kubernetes.io/projected/cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1-kube-api-access-98flq\") pod \"payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc\" (UID: \"cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1\") " pod="openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc" Nov 03 02:15:37.361262 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.358814 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc\" (UID: \"cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1\") " pod="openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc" Nov 03 02:15:37.373278 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.373179 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc\" (UID: \"cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1\") " pod="openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc" Nov 03 02:15:37.379251 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.379221 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4d6g\" (UniqueName: \"kubernetes.io/projected/1941d760-2cbe-42ba-a119-d1e4006d3c86-kube-api-access-l4d6g\") pod \"payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6\" (UID: \"1941d760-2cbe-42ba-a119-d1e4006d3c86\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6" Nov 03 02:15:37.405843 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.405813 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6" Nov 03 02:15:37.419073 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.419043 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98flq\" (UniqueName: \"kubernetes.io/projected/cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1-kube-api-access-98flq\") pod \"payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc\" (UID: \"cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1\") " pod="openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc" Nov 03 02:15:37.488835 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.488804 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc" Nov 03 02:15:37.527004 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.526860 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:15:37.527004 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.526964 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:15:37.527185 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.527011 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:15:37.531232 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.530416 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc"] Nov 03 02:15:37.781429 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:15:37.781397 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1941d760_2cbe_42ba_a119_d1e4006d3c86.slice/crio-d3673b8a9203b0d610631713f16c332bd8c9c9b8bddcc75dabe7b5bad1dd6a1e WatchSource:0}: Error finding container d3673b8a9203b0d610631713f16c332bd8c9c9b8bddcc75dabe7b5bad1dd6a1e: Status 404 returned error can't find the container with id d3673b8a9203b0d610631713f16c332bd8c9c9b8bddcc75dabe7b5bad1dd6a1e Nov 03 02:15:37.785536 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.785511 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6"] Nov 03 02:15:37.880052 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:15:37.880020 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb37d5d5_ece9_4fc1_8c06_62e85a3e1ec1.slice/crio-11a0d7ab6e3f76982f54f19b170c383f72c0a818d9429c10026a49f449d6ca81 WatchSource:0}: Error finding container 11a0d7ab6e3f76982f54f19b170c383f72c0a818d9429c10026a49f449d6ca81: Status 404 returned error can't find the container with id 11a0d7ab6e3f76982f54f19b170c383f72c0a818d9429c10026a49f449d6ca81 Nov 03 02:15:37.883273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:37.883249 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc"] Nov 03 02:15:38.131351 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:38.131259 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6" event={"ID":"1941d760-2cbe-42ba-a119-d1e4006d3c86","Type":"ContainerStarted","Data":"eb3e3694e1aaae4d863533c0310666e56ab9adac2ec99478edcaa0fc8c4b56ab"} Nov 03 02:15:38.131499 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:38.131358 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6" event={"ID":"1941d760-2cbe-42ba-a119-d1e4006d3c86","Type":"ContainerStarted","Data":"d3673b8a9203b0d610631713f16c332bd8c9c9b8bddcc75dabe7b5bad1dd6a1e"} Nov 03 02:15:38.131571 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:38.131533 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6" Nov 03 02:15:38.133272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:38.133241 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" event={"ID":"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb","Type":"ContainerStarted","Data":"4503c097403c59562ca7ae1334e8a7617af7cfe4f49c02a73c63404faedcfffc"} Nov 03 02:15:38.133396 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:38.133297 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" event={"ID":"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb","Type":"ContainerStarted","Data":"77157e4ffc0865b110b39afd465dc79032f516815369b86932167e4697e414aa"} Nov 03 02:15:38.135009 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:38.134983 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc" event={"ID":"cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1","Type":"ContainerStarted","Data":"4eac6f7dc6aa6cc0eeab74e359b5b32ff2ed32b403b51d7556ff5fb2315fa635"} Nov 03 02:15:38.135093 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:38.135034 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc" Nov 03 02:15:38.135093 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:38.135051 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc" event={"ID":"cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1","Type":"ContainerStarted","Data":"11a0d7ab6e3f76982f54f19b170c383f72c0a818d9429c10026a49f449d6ca81"} Nov 03 02:15:38.272205 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:38.272125 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6" podStartSLOduration=2.2721020960000002 podStartE2EDuration="2.272102096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:38.246385854 +0000 UTC m=+1419.930495010" watchObservedRunningTime="2026-11-03 02:15:38.272102096 +0000 UTC m=+1419.956211252" Nov 03 02:15:38.488187 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:38.488129 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc" podStartSLOduration=1.488113622 podStartE2EDuration="1.488113622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:38.48781761 +0000 UTC m=+1420.171926765" watchObservedRunningTime="2026-11-03 02:15:38.488113622 +0000 UTC m=+1420.172222819" Nov 03 02:15:38.489521 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:38.489454 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" podStartSLOduration=2.489439589 podStartE2EDuration="2.489439589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:38.373675959 +0000 UTC m=+1420.057785116" watchObservedRunningTime="2026-11-03 02:15:38.489439589 +0000 UTC m=+1420.173548745" Nov 03 02:15:38.749211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:38.749010 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:38.770215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:38.763242 5463 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 02:15:38.770215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:38.763308 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" podUID="9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 02:15:39.747104 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:39.747074 5463 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 02:15:39.747629 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:39.747124 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" podUID="9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 02:15:40.074830 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:40.074073 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:40.751237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:40.747561 5463 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 02:15:40.751237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:40.747622 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" podUID="9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 02:15:41.770211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:41.769257 5463 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 02:15:41.770211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:41.769315 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" podUID="9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 02:15:42.769216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:42.766287 5463 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 02:15:42.769216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:42.766361 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" podUID="9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 02:15:43.008655 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:15:43.008623 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:15:43.747868 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:43.747835 5463 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 02:15:43.748037 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:43.747910 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" podUID="9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 02:15:44.942436 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:44.942403 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:45.076458 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:45.076364 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd"] Nov 03 02:15:45.085176 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:45.085119 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:45.085987 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:45.085939 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:45.087164 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:45.087121 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:15:45.101015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:45.100290 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab7db544-56wpd"] Nov 03 02:15:45.163317 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:45.163284 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:45.302116 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:45.302019 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:46.504213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:46.496752 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4233154b-1891-4123-8c65-c5e910c1ab9a" path="/var/lib/kubelet/pods/4233154b-1891-4123-8c65-c5e910c1ab9a/volumes" Nov 03 02:15:49.049582 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:49.049549 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts"] Nov 03 02:15:49.099399 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:49.099368 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eda9adc-cntts"] Nov 03 02:15:49.147035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:49.147007 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc" Nov 03 02:15:49.147173 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:49.147072 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6" Nov 03 02:15:49.633385 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:49.633350 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:50.444434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:50.444403 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06609e21-7e29-454a-a6ad-4dced184ed62" path="/var/lib/kubelet/pods/06609e21-7e29-454a-a6ad-4dced184ed62/volumes" Nov 03 02:15:54.432719 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:54.432670 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:59.175692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:15:59.175556 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:02.048910 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:02.048877 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4"] Nov 03 02:16:02.098899 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:02.098863 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf59c59a-m8hd4"] Nov 03 02:16:02.444244 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:02.444215 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b54649d6-2925-4feb-9f3f-ca3a91cec3ff" path="/var/lib/kubelet/pods/b54649d6-2925-4feb-9f3f-ca3a91cec3ff/volumes" Nov 03 02:16:03.626970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:03.626939 5463 scope.go:122] "RemoveContainer" containerID="9f499029bbce2c2ec0de400dd61f1b632396644b908a6b9c3a36b808591532c7" Nov 03 02:16:03.637500 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:03.637475 5463 scope.go:122] "RemoveContainer" containerID="24c9c5f9633337e7c66fdf5652b82bd7fe96d61126dbdf8d717c4e93b996a2c3" Nov 03 02:16:03.878762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:03.878670 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:04.038912 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:04.038880 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:04.185499 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:04.185451 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:08.051153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:08.051112 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4"] Nov 03 02:16:08.101493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:08.101461 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd336ebf-rdzq4"] Nov 03 02:16:08.447060 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:08.447026 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b17eb9a-2379-441c-8d88-3b6d1148462c" path="/var/lib/kubelet/pods/2b17eb9a-2379-441c-8d88-3b6d1148462c/volumes" Nov 03 02:16:09.265616 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:09.265539 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:09.409645 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:09.409608 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:11.314326 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:11.314288 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:13.021847 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:16:13.021800 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:16:13.051411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:13.051380 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:14.126387 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:14.126350 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:16.454673 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:16.454639 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck"] Nov 03 02:16:16.564791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:16.564757 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b"] Nov 03 02:16:16.833610 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:16.833512 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:17.033723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:17.033688 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:17.450851 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:17.450815 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:18.084826 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:18.084790 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6"] Nov 03 02:16:18.107840 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:18.107802 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4fa141e-6h4p6"] Nov 03 02:16:18.466330 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:18.466293 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d7258ba-d952-4fd7-8e9f-9d1cae62de09" path="/var/lib/kubelet/pods/1d7258ba-d952-4fd7-8e9f-9d1cae62de09/volumes" Nov 03 02:16:19.257362 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:19.257327 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:21.455308 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:21.455250 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck" podUID="8bd6ab07-9430-466d-9162-8e89ace4f015" containerName="payload-processing" containerID="cri-o://f95354f7969a841134adb8f6b0068362d88f27e2254adf4fc33dad10febeb0ea" gracePeriod=25 Nov 03 02:16:21.565315 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:21.565271 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b" podUID="490ca339-509d-4f76-b2b7-6e6f57be24c7" containerName="payload-pre-processing" containerID="cri-o://7de0b497e3f7eab0e36890ea96e067bc27cf75dc16c7fad9fe10c42cf83fd6ba" gracePeriod=25 Nov 03 02:16:21.576696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:21.576666 5463 generic.go:356] "Generic (PLEG): container finished" podID="8bd6ab07-9430-466d-9162-8e89ace4f015" containerID="f95354f7969a841134adb8f6b0068362d88f27e2254adf4fc33dad10febeb0ea" exitCode=0 Nov 03 02:16:21.576808 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:21.576737 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck" event={"ID":"8bd6ab07-9430-466d-9162-8e89ace4f015","Type":"ContainerDied","Data":"f95354f7969a841134adb8f6b0068362d88f27e2254adf4fc33dad10febeb0ea"} Nov 03 02:16:21.799278 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:21.799248 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck" Nov 03 02:16:21.930209 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:21.930164 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bd6ab07-9430-466d-9162-8e89ace4f015-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd6ab07-9430-466d-9162-8e89ace4f015-plugins-config-volume\") pod \"8bd6ab07-9430-466d-9162-8e89ace4f015\" (UID: \"8bd6ab07-9430-466d-9162-8e89ace4f015\") " Nov 03 02:16:21.939587 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:21.930797 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd6ab07-9430-466d-9162-8e89ace4f015-plugins-config-volume" pod "8bd6ab07-9430-466d-9162-8e89ace4f015" (UID: "8bd6ab07-9430-466d-9162-8e89ace4f015"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:16:21.939587 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:21.933053 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b" Nov 03 02:16:21.939826 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:21.939810 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd6ab07-9430-466d-9162-8e89ace4f015-kube-api-access-9ssbx\" (UniqueName: \"kubernetes.io/projected/8bd6ab07-9430-466d-9162-8e89ace4f015-kube-api-access-9ssbx\") pod \"8bd6ab07-9430-466d-9162-8e89ace4f015\" (UID: \"8bd6ab07-9430-466d-9162-8e89ace4f015\") " Nov 03 02:16:21.940483 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:21.940457 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd6ab07-9430-466d-9162-8e89ace4f015-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:21.942082 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:21.942055 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd6ab07-9430-466d-9162-8e89ace4f015-kube-api-access-9ssbx" pod "8bd6ab07-9430-466d-9162-8e89ace4f015" (UID: "8bd6ab07-9430-466d-9162-8e89ace4f015"). InnerVolumeSpecName "kube-api-access-9ssbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:22.040997 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.040920 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/490ca339-509d-4f76-b2b7-6e6f57be24c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490ca339-509d-4f76-b2b7-6e6f57be24c7-plugins-config-volume\") pod \"490ca339-509d-4f76-b2b7-6e6f57be24c7\" (UID: \"490ca339-509d-4f76-b2b7-6e6f57be24c7\") " Nov 03 02:16:22.041143 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.041134 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/490ca339-509d-4f76-b2b7-6e6f57be24c7-kube-api-access-xjfvt\" (UniqueName: \"kubernetes.io/projected/490ca339-509d-4f76-b2b7-6e6f57be24c7-kube-api-access-xjfvt\") pod \"490ca339-509d-4f76-b2b7-6e6f57be24c7\" (UID: \"490ca339-509d-4f76-b2b7-6e6f57be24c7\") " Nov 03 02:16:22.041333 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.041311 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/490ca339-509d-4f76-b2b7-6e6f57be24c7-plugins-config-volume" pod "490ca339-509d-4f76-b2b7-6e6f57be24c7" (UID: "490ca339-509d-4f76-b2b7-6e6f57be24c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:16:22.041708 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.041690 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ssbx\" (UniqueName: \"kubernetes.io/projected/8bd6ab07-9430-466d-9162-8e89ace4f015-kube-api-access-9ssbx\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:22.041708 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.041708 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490ca339-509d-4f76-b2b7-6e6f57be24c7-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:22.043237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.043215 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490ca339-509d-4f76-b2b7-6e6f57be24c7-kube-api-access-xjfvt" pod "490ca339-509d-4f76-b2b7-6e6f57be24c7" (UID: "490ca339-509d-4f76-b2b7-6e6f57be24c7"). InnerVolumeSpecName "kube-api-access-xjfvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:22.143103 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.143071 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjfvt\" (UniqueName: \"kubernetes.io/projected/490ca339-509d-4f76-b2b7-6e6f57be24c7-kube-api-access-xjfvt\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:22.586984 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.586949 5463 generic.go:356] "Generic (PLEG): container finished" podID="490ca339-509d-4f76-b2b7-6e6f57be24c7" containerID="7de0b497e3f7eab0e36890ea96e067bc27cf75dc16c7fad9fe10c42cf83fd6ba" exitCode=0 Nov 03 02:16:22.587493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.586999 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b" Nov 03 02:16:22.587493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.587042 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b" event={"ID":"490ca339-509d-4f76-b2b7-6e6f57be24c7","Type":"ContainerDied","Data":"7de0b497e3f7eab0e36890ea96e067bc27cf75dc16c7fad9fe10c42cf83fd6ba"} Nov 03 02:16:22.587493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.587118 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b" event={"ID":"490ca339-509d-4f76-b2b7-6e6f57be24c7","Type":"ContainerDied","Data":"b1c5ffe54722e1b16d3e1a2d05de96fb616e03e7382f571b6e12ce9850a7ad91"} Nov 03 02:16:22.587493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.587148 5463 scope.go:122] "RemoveContainer" containerID="7de0b497e3f7eab0e36890ea96e067bc27cf75dc16c7fad9fe10c42cf83fd6ba" Nov 03 02:16:22.589028 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.588886 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck" event={"ID":"8bd6ab07-9430-466d-9162-8e89ace4f015","Type":"ContainerDied","Data":"d706282e4e286d7584ea345c309fda4cce313c347e9bf738fdc2b80729d7d343"} Nov 03 02:16:22.589028 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.588911 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck" Nov 03 02:16:22.598526 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.598508 5463 scope.go:122] "RemoveContainer" containerID="7de0b497e3f7eab0e36890ea96e067bc27cf75dc16c7fad9fe10c42cf83fd6ba" Nov 03 02:16:22.598809 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:16:22.598789 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7de0b497e3f7eab0e36890ea96e067bc27cf75dc16c7fad9fe10c42cf83fd6ba\": container with ID starting with 7de0b497e3f7eab0e36890ea96e067bc27cf75dc16c7fad9fe10c42cf83fd6ba not found: ID does not exist" containerID="7de0b497e3f7eab0e36890ea96e067bc27cf75dc16c7fad9fe10c42cf83fd6ba" Nov 03 02:16:22.598877 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.598815 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de0b497e3f7eab0e36890ea96e067bc27cf75dc16c7fad9fe10c42cf83fd6ba"} err="failed to get container status \"7de0b497e3f7eab0e36890ea96e067bc27cf75dc16c7fad9fe10c42cf83fd6ba\": rpc error: code = NotFound desc = could not find container \"7de0b497e3f7eab0e36890ea96e067bc27cf75dc16c7fad9fe10c42cf83fd6ba\": container with ID starting with 7de0b497e3f7eab0e36890ea96e067bc27cf75dc16c7fad9fe10c42cf83fd6ba not found: ID does not exist" Nov 03 02:16:22.598877 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.598829 5463 scope.go:122] "RemoveContainer" containerID="f95354f7969a841134adb8f6b0068362d88f27e2254adf4fc33dad10febeb0ea" Nov 03 02:16:22.781350 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.780495 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck"] Nov 03 02:16:22.836390 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.836356 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7f332d-5d6f7968b9-fk8ck"] Nov 03 02:16:22.888702 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.888638 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b"] Nov 03 02:16:22.895070 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:22.895012 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7f332d-cbcdfb647-w6w2b"] Nov 03 02:16:23.302348 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:23.302318 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:23.995219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:23.994950 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:24.137265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:24.137231 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:24.286034 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:24.285931 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:24.444659 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:24.444626 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="490ca339-509d-4f76-b2b7-6e6f57be24c7" path="/var/lib/kubelet/pods/490ca339-509d-4f76-b2b7-6e6f57be24c7/volumes" Nov 03 02:16:24.445185 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:24.445169 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bd6ab07-9430-466d-9162-8e89ace4f015" path="/var/lib/kubelet/pods/8bd6ab07-9430-466d-9162-8e89ace4f015/volumes" Nov 03 02:16:26.047379 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:26.047349 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h"] Nov 03 02:16:26.097278 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:26.097249 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aa43ebe-rmm7h"] Nov 03 02:16:26.444900 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:26.444864 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28dab9fe-c0a9-465c-98ac-759f60d92cbf" path="/var/lib/kubelet/pods/28dab9fe-c0a9-465c-98ac-759f60d92cbf/volumes" Nov 03 02:16:28.929339 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:28.929306 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:33.749481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:33.749445 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:37.910111 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:37.910071 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:39.628578 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:39.628542 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:43.010425 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:16:43.010383 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:16:44.331560 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:44.331526 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:46.360681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:46.360646 5463 patch_prober.go:29] interesting pod/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": read tcp 10.133.0.2:58842->10.133.0.128:15021: read: connection reset by peer" start-of-body= Nov 03 02:16:46.361056 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:46.360728 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" podUID="2fd41e7e-553d-45c9-b25c-d4aa074c899d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": read tcp 10.133.0.2:58842->10.133.0.128:15021: read: connection reset by peer" Nov 03 02:16:46.618046 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:46.617966 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:46.780828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:46.780795 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:46.945064 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:46.945020 5463 generic.go:356] "Generic (PLEG): container finished" podID="2fd41e7e-553d-45c9-b25c-d4aa074c899d" containerID="537e46e20db1abb465a30d70c651246ee409315f3744de348bd838ffee9dfb5a" exitCode=0 Nov 03 02:16:46.945272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:46.945092 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" event={"ID":"2fd41e7e-553d-45c9-b25c-d4aa074c899d","Type":"ContainerDied","Data":"537e46e20db1abb465a30d70c651246ee409315f3744de348bd838ffee9dfb5a"} Nov 03 02:16:46.945556 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:46.945540 5463 scope.go:122] "RemoveContainer" containerID="537e46e20db1abb465a30d70c651246ee409315f3744de348bd838ffee9dfb5a" Nov 03 02:16:46.947131 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:46.947101 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:16:46.947231 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:46.947170 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:16:46.947231 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:46.947225 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:16:47.965050 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:47.965018 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" event={"ID":"2fd41e7e-553d-45c9-b25c-d4aa074c899d","Type":"ContainerStarted","Data":"db22fc899b5d467fc88580f4fa82cc84e10d1b608b5602a0c266894e8399eedf"} Nov 03 02:16:48.085803 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:48.085751 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7"] Nov 03 02:16:48.086071 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:48.086011 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" podUID="88714959-afe9-4294-b124-fca1f9d0c4f3" containerName="istio-proxy" containerID="cri-o://f7907928918c239aa4669ee8f9aa050a50436457c7e0181fa72319d611a6eae5" gracePeriod=30 Nov 03 02:16:48.092091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:48.092058 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5"] Nov 03 02:16:48.092303 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:48.092278 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" podUID="731a025d-7302-4a22-8b43-70f96d8f05bb" containerName="istio-proxy" containerID="cri-o://28fd5f6ba5091a131cb357b2f1635f94929e6ec12bcb32112d8adcff51847e90" gracePeriod=30 Nov 03 02:16:48.127446 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:48.127409 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx"] Nov 03 02:16:48.127853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:48.127823 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" podUID="2617013b-bed0-446d-a597-31bb323d1c6a" containerName="istio-proxy" containerID="cri-o://d47147148bca0b49acfcb877762bf35db07ee00784f8854099753e9fc5625abf" gracePeriod=30 Nov 03 02:16:48.132502 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:48.132476 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf"] Nov 03 02:16:48.132656 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:48.132636 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" podUID="dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" containerName="istio-proxy" containerID="cri-o://01c1dd81d2ac1849ef17d16215b7b100b407a0698d4d277555af1db42e47d994" gracePeriod=30 Nov 03 02:16:48.194213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:48.188657 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:16:48.194213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:48.188774 5463 patch_prober.go:29] interesting pod/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 02:16:48.194213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:48.188813 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" podUID="2fd41e7e-553d-45c9-b25c-d4aa074c899d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 02:16:48.257957 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:48.257728 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:48.718228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:48.711330 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n"] Nov 03 02:16:48.718228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:48.711574 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" podUID="9ec04276-3f4d-4bdf-96bf-37f93fd0913e" containerName="istio-proxy" containerID="cri-o://8649bcda78befe6cb20fc947e18bfc41f77ed6a468eb1bd07e501eb688529f8d" gracePeriod=30 Nov 03 02:16:48.767560 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:48.767383 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:49.182775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:49.182285 5463 patch_prober.go:29] interesting pod/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 02:16:49.182775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:49.182366 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" podUID="2fd41e7e-553d-45c9-b25c-d4aa074c899d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 02:16:49.277252 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:49.276510 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:49.283933 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:49.283815 5463 patch_prober.go:29] interesting pod/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:16:49.283933 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:49.283882 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" podUID="9ec04276-3f4d-4bdf-96bf-37f93fd0913e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:16:49.414389 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:49.413150 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7"] Nov 03 02:16:49.414389 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:49.413400 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" podUID="13e969fe-f189-40f2-b929-e462f6d4b86b" containerName="istio-proxy" containerID="cri-o://141f045aa4d400f57e8e3bf8e8ce8f30fe30ae85ee65e7fce4e42303090a2003" gracePeriod=30 Nov 03 02:16:49.417658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:49.417346 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8"] Nov 03 02:16:49.417658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:49.417504 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" podUID="cc2f2fb4-5a6f-4778-90e5-9a857204f663" containerName="istio-proxy" containerID="cri-o://35464c1f7916006f4cedd3f0b5ba10a46959f8a5e4b006e1097fe0ee2a577f0e" gracePeriod=30 Nov 03 02:16:49.422217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:49.421285 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq"] Nov 03 02:16:49.422217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:49.421452 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" podUID="32ef5394-19c0-4b33-84f6-2087dede03a0" containerName="istio-proxy" containerID="cri-o://0cfba01290b039c4902ed6b049e39f9411f18fc8eb7fa24bb60395651e1e8792" gracePeriod=30 Nov 03 02:16:49.425734 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:49.425713 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p"] Nov 03 02:16:49.426049 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:49.426015 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" podUID="2746f922-b469-4383-90bb-3be08f9256d7" containerName="istio-proxy" containerID="cri-o://cb15eb4b66f3990c6f574a0ab10364eee39b9a466f6aeaa245dbdd7e1dc8942c" gracePeriod=30 Nov 03 02:16:49.583350 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:49.574870 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:50.068876 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:50.068841 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62"] Nov 03 02:16:50.087152 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:50.087111 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ddcb09b-mwm62"] Nov 03 02:16:50.184216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:50.180490 5463 patch_prober.go:29] interesting pod/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 02:16:50.184216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:50.180915 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" podUID="2fd41e7e-553d-45c9-b25c-d4aa074c899d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 02:16:50.444204 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:50.444164 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d4d6abf-5729-4728-b755-64c2231dd9bf" path="/var/lib/kubelet/pods/3d4d6abf-5729-4728-b755-64c2231dd9bf/volumes" Nov 03 02:16:50.603527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:50.603488 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:16:50.603695 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:50.603562 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" podUID="2746f922-b469-4383-90bb-3be08f9256d7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:16:51.180216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:51.179689 5463 patch_prober.go:29] interesting pod/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 02:16:51.180216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:51.179747 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" podUID="2fd41e7e-553d-45c9-b25c-d4aa074c899d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 02:16:52.182217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:52.180574 5463 patch_prober.go:29] interesting pod/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 02:16:52.182217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:52.180652 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" podUID="2fd41e7e-553d-45c9-b25c-d4aa074c899d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 02:16:52.981216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:52.977725 5463 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:16:52.981216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:52.977784 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" podUID="dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:16:53.180976 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.180532 5463 patch_prober.go:29] interesting pod/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 02:16:53.180976 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.180593 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" podUID="2fd41e7e-553d-45c9-b25c-d4aa074c899d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 02:16:53.504795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.503997 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:16:53.522755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.522719 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-workload-socket\") pod \"88714959-afe9-4294-b124-fca1f9d0c4f3\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " Nov 03 02:16:53.522916 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.522775 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-credential-socket\") pod \"88714959-afe9-4294-b124-fca1f9d0c4f3\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " Nov 03 02:16:53.522916 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.522845 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-workload-certs\") pod \"88714959-afe9-4294-b124-fca1f9d0c4f3\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " Nov 03 02:16:53.522916 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.522885 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-envoy\") pod \"88714959-afe9-4294-b124-fca1f9d0c4f3\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " Nov 03 02:16:53.523090 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.522924 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88714959-afe9-4294-b124-fca1f9d0c4f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88714959-afe9-4294-b124-fca1f9d0c4f3-istiod-ca-cert\") pod \"88714959-afe9-4294-b124-fca1f9d0c4f3\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " Nov 03 02:16:53.523090 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.523076 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-podinfo\") pod \"88714959-afe9-4294-b124-fca1f9d0c4f3\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " Nov 03 02:16:53.523349 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.523225 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88714959-afe9-4294-b124-fca1f9d0c4f3-kube-api-access-95vlf\" (UniqueName: \"kubernetes.io/projected/88714959-afe9-4294-b124-fca1f9d0c4f3-kube-api-access-95vlf\") pod \"88714959-afe9-4294-b124-fca1f9d0c4f3\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " Nov 03 02:16:53.523349 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.523273 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-token\" (UniqueName: \"kubernetes.io/projected/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-token\") pod \"88714959-afe9-4294-b124-fca1f9d0c4f3\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " Nov 03 02:16:53.523349 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.523338 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-data\") pod \"88714959-afe9-4294-b124-fca1f9d0c4f3\" (UID: \"88714959-afe9-4294-b124-fca1f9d0c4f3\") " Nov 03 02:16:53.525212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.524902 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-workload-socket" pod "88714959-afe9-4294-b124-fca1f9d0c4f3" (UID: "88714959-afe9-4294-b124-fca1f9d0c4f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:53.525212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.525107 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-credential-socket" pod "88714959-afe9-4294-b124-fca1f9d0c4f3" (UID: "88714959-afe9-4294-b124-fca1f9d0c4f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:53.525402 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.525302 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-workload-certs" pod "88714959-afe9-4294-b124-fca1f9d0c4f3" (UID: "88714959-afe9-4294-b124-fca1f9d0c4f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:53.535590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.525761 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-data" pod "88714959-afe9-4294-b124-fca1f9d0c4f3" (UID: "88714959-afe9-4294-b124-fca1f9d0c4f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:53.535590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.526056 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88714959-afe9-4294-b124-fca1f9d0c4f3-istiod-ca-cert" pod "88714959-afe9-4294-b124-fca1f9d0c4f3" (UID: "88714959-afe9-4294-b124-fca1f9d0c4f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:16:53.546234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.545231 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-podinfo" pod "88714959-afe9-4294-b124-fca1f9d0c4f3" (UID: "88714959-afe9-4294-b124-fca1f9d0c4f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:16:53.546234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.545229 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-envoy" pod "88714959-afe9-4294-b124-fca1f9d0c4f3" (UID: "88714959-afe9-4294-b124-fca1f9d0c4f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:53.546234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.545380 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88714959-afe9-4294-b124-fca1f9d0c4f3-kube-api-access-95vlf" pod "88714959-afe9-4294-b124-fca1f9d0c4f3" (UID: "88714959-afe9-4294-b124-fca1f9d0c4f3"). InnerVolumeSpecName "kube-api-access-95vlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:53.554716 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.554675 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-token" pod "88714959-afe9-4294-b124-fca1f9d0c4f3" (UID: "88714959-afe9-4294-b124-fca1f9d0c4f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:53.626885 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.626672 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.626885 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.626715 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95vlf\" (UniqueName: \"kubernetes.io/projected/88714959-afe9-4294-b124-fca1f9d0c4f3-kube-api-access-95vlf\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.626885 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.626734 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.626885 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.626749 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.626885 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.626764 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.626885 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.626779 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.626885 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.626798 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.626885 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.626814 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88714959-afe9-4294-b124-fca1f9d0c4f3-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.626885 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.626831 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88714959-afe9-4294-b124-fca1f9d0c4f3-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.731303 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.731276 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:16:53.735969 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.735945 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:16:53.740767 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.740746 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:16:53.830177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.830083 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-workload-certs\") pod \"2617013b-bed0-446d-a597-31bb323d1c6a\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " Nov 03 02:16:53.830540 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.830516 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-workload-socket\") pod \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " Nov 03 02:16:53.830698 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.830682 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-data\") pod \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " Nov 03 02:16:53.830825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.830811 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2617013b-bed0-446d-a597-31bb323d1c6a-kube-api-access-j55js\" (UniqueName: \"kubernetes.io/projected/2617013b-bed0-446d-a597-31bb323d1c6a-kube-api-access-j55js\") pod \"2617013b-bed0-446d-a597-31bb323d1c6a\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " Nov 03 02:16:53.830934 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.830921 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-istio-data\") pod \"2617013b-bed0-446d-a597-31bb323d1c6a\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " Nov 03 02:16:53.831092 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.831074 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731a025d-7302-4a22-8b43-70f96d8f05bb-kube-api-access-xhs98\" (UniqueName: \"kubernetes.io/projected/731a025d-7302-4a22-8b43-70f96d8f05bb-kube-api-access-xhs98\") pod \"731a025d-7302-4a22-8b43-70f96d8f05bb\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " Nov 03 02:16:53.831225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.831211 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-podinfo\") pod \"731a025d-7302-4a22-8b43-70f96d8f05bb\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " Nov 03 02:16:53.832063 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.832044 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-workload-socket\") pod \"2617013b-bed0-446d-a597-31bb323d1c6a\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " Nov 03 02:16:53.832285 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.830747 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-workload-socket" pod "dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" (UID: "dfeb7b50-ad6f-4fa2-b672-cb47242deb1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:53.832417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.830921 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-data" pod "dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" (UID: "dfeb7b50-ad6f-4fa2-b672-cb47242deb1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:53.832417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.831270 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-workload-certs" pod "2617013b-bed0-446d-a597-31bb323d1c6a" (UID: "2617013b-bed0-446d-a597-31bb323d1c6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:53.832417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.831521 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-istio-data" pod "2617013b-bed0-446d-a597-31bb323d1c6a" (UID: "2617013b-bed0-446d-a597-31bb323d1c6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:53.832417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.832326 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-workload-socket" pod "2617013b-bed0-446d-a597-31bb323d1c6a" (UID: "2617013b-bed0-446d-a597-31bb323d1c6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:53.832682 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.832665 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-podinfo\") pod \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " Nov 03 02:16:53.832846 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.832830 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/731a025d-7302-4a22-8b43-70f96d8f05bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/731a025d-7302-4a22-8b43-70f96d8f05bb-istiod-ca-cert\") pod \"731a025d-7302-4a22-8b43-70f96d8f05bb\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " Nov 03 02:16:53.834018 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.833997 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istiod-ca-cert\") pod \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.834205 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-token\" (UniqueName: \"kubernetes.io/projected/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-token\") pod \"731a025d-7302-4a22-8b43-70f96d8f05bb\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.834256 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-envoy\") pod \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.834318 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-data\") pod \"731a025d-7302-4a22-8b43-70f96d8f05bb\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.834355 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-workload-certs\") pod \"731a025d-7302-4a22-8b43-70f96d8f05bb\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.834431 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-credential-socket\") pod \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.834468 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-credential-socket\") pod \"731a025d-7302-4a22-8b43-70f96d8f05bb\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.834499 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-token\" (UniqueName: \"kubernetes.io/projected/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-token\") pod \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.834528 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2617013b-bed0-446d-a597-31bb323d1c6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2617013b-bed0-446d-a597-31bb323d1c6a-istiod-ca-cert\") pod \"2617013b-bed0-446d-a597-31bb323d1c6a\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.834569 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2617013b-bed0-446d-a597-31bb323d1c6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2617013b-bed0-446d-a597-31bb323d1c6a-istio-podinfo\") pod \"2617013b-bed0-446d-a597-31bb323d1c6a\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.834627 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-envoy\") pod \"731a025d-7302-4a22-8b43-70f96d8f05bb\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.834850 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2617013b-bed0-446d-a597-31bb323d1c6a-istio-token\" (UniqueName: \"kubernetes.io/projected/2617013b-bed0-446d-a597-31bb323d1c6a-istio-token\") pod \"2617013b-bed0-446d-a597-31bb323d1c6a\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.834907 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-workload-certs\") pod \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.834997 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-kube-api-access-7zzjg\" (UniqueName: \"kubernetes.io/projected/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-kube-api-access-7zzjg\") pod \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\" (UID: \"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e\") " Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.835040 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-credential-socket\") pod \"2617013b-bed0-446d-a597-31bb323d1c6a\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.835092 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-istio-envoy\") pod \"2617013b-bed0-446d-a597-31bb323d1c6a\" (UID: \"2617013b-bed0-446d-a597-31bb323d1c6a\") " Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.835128 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-workload-socket\") pod \"731a025d-7302-4a22-8b43-70f96d8f05bb\" (UID: \"731a025d-7302-4a22-8b43-70f96d8f05bb\") " Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.833328 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2617013b-bed0-446d-a597-31bb323d1c6a-kube-api-access-j55js" pod "2617013b-bed0-446d-a597-31bb323d1c6a" (UID: "2617013b-bed0-446d-a597-31bb323d1c6a"). InnerVolumeSpecName "kube-api-access-j55js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.833526 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/731a025d-7302-4a22-8b43-70f96d8f05bb-istiod-ca-cert" pod "731a025d-7302-4a22-8b43-70f96d8f05bb" (UID: "731a025d-7302-4a22-8b43-70f96d8f05bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.835294 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-credential-socket" pod "731a025d-7302-4a22-8b43-70f96d8f05bb" (UID: "731a025d-7302-4a22-8b43-70f96d8f05bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.835521 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731a025d-7302-4a22-8b43-70f96d8f05bb-kube-api-access-xhs98" pod "731a025d-7302-4a22-8b43-70f96d8f05bb" (UID: "731a025d-7302-4a22-8b43-70f96d8f05bb"). InnerVolumeSpecName "kube-api-access-xhs98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.835534 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-data" pod "731a025d-7302-4a22-8b43-70f96d8f05bb" (UID: "731a025d-7302-4a22-8b43-70f96d8f05bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.835630 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-workload-certs" pod "731a025d-7302-4a22-8b43-70f96d8f05bb" (UID: "731a025d-7302-4a22-8b43-70f96d8f05bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:53.836119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.835782 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-credential-socket" pod "dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" (UID: "dfeb7b50-ad6f-4fa2-b672-cb47242deb1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:53.837469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.836159 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-workload-certs" pod "dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" (UID: "dfeb7b50-ad6f-4fa2-b672-cb47242deb1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:53.837469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.836174 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istiod-ca-cert" pod "dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" (UID: "dfeb7b50-ad6f-4fa2-b672-cb47242deb1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:16:53.837469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.836342 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2617013b-bed0-446d-a597-31bb323d1c6a-istiod-ca-cert" pod "2617013b-bed0-446d-a597-31bb323d1c6a" (UID: "2617013b-bed0-446d-a597-31bb323d1c6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:16:53.837469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.836401 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-podinfo" pod "dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" (UID: "dfeb7b50-ad6f-4fa2-b672-cb47242deb1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:16:53.837469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.836977 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-credential-socket" pod "2617013b-bed0-446d-a597-31bb323d1c6a" (UID: "2617013b-bed0-446d-a597-31bb323d1c6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:53.837746 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.837656 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.837746 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.837679 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j55js\" (UniqueName: \"kubernetes.io/projected/2617013b-bed0-446d-a597-31bb323d1c6a-kube-api-access-j55js\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.837746 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.837693 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.837746 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.837710 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhs98\" (UniqueName: \"kubernetes.io/projected/731a025d-7302-4a22-8b43-70f96d8f05bb-kube-api-access-xhs98\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.837746 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.837726 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.837746 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.837739 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.838147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.837752 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/731a025d-7302-4a22-8b43-70f96d8f05bb-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.838147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.837768 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.838147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.837781 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.838147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.837794 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.838147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.837809 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.838147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.837821 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.838147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.837835 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2617013b-bed0-446d-a597-31bb323d1c6a-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.838147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.837849 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.838147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.837863 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.838147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.837875 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.838147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.837889 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.838147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.837956 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-workload-socket" pod "731a025d-7302-4a22-8b43-70f96d8f05bb" (UID: "731a025d-7302-4a22-8b43-70f96d8f05bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:53.838753 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.838151 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-envoy" pod "731a025d-7302-4a22-8b43-70f96d8f05bb" (UID: "731a025d-7302-4a22-8b43-70f96d8f05bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:53.840859 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.840836 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-kube-api-access-7zzjg" pod "dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" (UID: "dfeb7b50-ad6f-4fa2-b672-cb47242deb1e"). InnerVolumeSpecName "kube-api-access-7zzjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:53.841457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.841406 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-podinfo" pod "731a025d-7302-4a22-8b43-70f96d8f05bb" (UID: "731a025d-7302-4a22-8b43-70f96d8f05bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:16:53.841559 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.841523 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2617013b-bed0-446d-a597-31bb323d1c6a-istio-podinfo" pod "2617013b-bed0-446d-a597-31bb323d1c6a" (UID: "2617013b-bed0-446d-a597-31bb323d1c6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:16:53.842079 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.842049 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-token" pod "731a025d-7302-4a22-8b43-70f96d8f05bb" (UID: "731a025d-7302-4a22-8b43-70f96d8f05bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:53.842174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.842155 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-envoy" pod "dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" (UID: "dfeb7b50-ad6f-4fa2-b672-cb47242deb1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:53.842529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.842488 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2617013b-bed0-446d-a597-31bb323d1c6a-istio-token" pod "2617013b-bed0-446d-a597-31bb323d1c6a" (UID: "2617013b-bed0-446d-a597-31bb323d1c6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:53.843438 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.843342 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-istio-envoy" pod "2617013b-bed0-446d-a597-31bb323d1c6a" (UID: "2617013b-bed0-446d-a597-31bb323d1c6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:53.843438 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.843411 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-token" pod "dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" (UID: "dfeb7b50-ad6f-4fa2-b672-cb47242deb1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:53.939784 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.939735 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.939784 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.939775 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.939784 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.939789 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.940023 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.939802 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.940023 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.939813 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2617013b-bed0-446d-a597-31bb323d1c6a-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.940023 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.939825 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.940023 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.939836 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2617013b-bed0-446d-a597-31bb323d1c6a-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.940023 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.939848 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zzjg\" (UniqueName: \"kubernetes.io/projected/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e-kube-api-access-7zzjg\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.940023 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.939862 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2617013b-bed0-446d-a597-31bb323d1c6a-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:53.940023 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:53.939876 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/731a025d-7302-4a22-8b43-70f96d8f05bb-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.077060 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.077030 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:16:54.119759 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.119656 5463 generic.go:356] "Generic (PLEG): container finished" podID="731a025d-7302-4a22-8b43-70f96d8f05bb" containerID="28fd5f6ba5091a131cb357b2f1635f94929e6ec12bcb32112d8adcff51847e90" exitCode=0 Nov 03 02:16:54.119759 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.119743 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" Nov 03 02:16:54.119991 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.119755 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" event={"ID":"731a025d-7302-4a22-8b43-70f96d8f05bb","Type":"ContainerDied","Data":"28fd5f6ba5091a131cb357b2f1635f94929e6ec12bcb32112d8adcff51847e90"} Nov 03 02:16:54.119991 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.119821 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5" event={"ID":"731a025d-7302-4a22-8b43-70f96d8f05bb","Type":"ContainerDied","Data":"95fe22851627fae52ba1e7ff045574d9706aea39a2cc9fc96ab1654775f8a3e4"} Nov 03 02:16:54.119991 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.119855 5463 scope.go:122] "RemoveContainer" containerID="28fd5f6ba5091a131cb357b2f1635f94929e6ec12bcb32112d8adcff51847e90" Nov 03 02:16:54.132630 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.132602 5463 generic.go:356] "Generic (PLEG): container finished" podID="9ec04276-3f4d-4bdf-96bf-37f93fd0913e" containerID="8649bcda78befe6cb20fc947e18bfc41f77ed6a468eb1bd07e501eb688529f8d" exitCode=0 Nov 03 02:16:54.132739 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.132649 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" Nov 03 02:16:54.132739 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.132679 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" event={"ID":"9ec04276-3f4d-4bdf-96bf-37f93fd0913e","Type":"ContainerDied","Data":"8649bcda78befe6cb20fc947e18bfc41f77ed6a468eb1bd07e501eb688529f8d"} Nov 03 02:16:54.132739 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.132728 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n" event={"ID":"9ec04276-3f4d-4bdf-96bf-37f93fd0913e","Type":"ContainerDied","Data":"4070552af0ef31f6818248a107c83fe8fcca9d370c960a37910b2724cfb6b6a3"} Nov 03 02:16:54.134553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.134528 5463 generic.go:356] "Generic (PLEG): container finished" podID="dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" containerID="01c1dd81d2ac1849ef17d16215b7b100b407a0698d4d277555af1db42e47d994" exitCode=0 Nov 03 02:16:54.134668 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.134575 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" Nov 03 02:16:54.134668 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.134614 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" event={"ID":"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e","Type":"ContainerDied","Data":"01c1dd81d2ac1849ef17d16215b7b100b407a0698d4d277555af1db42e47d994"} Nov 03 02:16:54.134798 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.134658 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf" event={"ID":"dfeb7b50-ad6f-4fa2-b672-cb47242deb1e","Type":"ContainerDied","Data":"87a2e7fd42de7d07a584ecb549b719aab25bc2e4bb5b84d6b57068c0475e7e82"} Nov 03 02:16:54.141319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.140783 5463 generic.go:356] "Generic (PLEG): container finished" podID="88714959-afe9-4294-b124-fca1f9d0c4f3" containerID="f7907928918c239aa4669ee8f9aa050a50436457c7e0181fa72319d611a6eae5" exitCode=0 Nov 03 02:16:54.141319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.140914 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" event={"ID":"88714959-afe9-4294-b124-fca1f9d0c4f3","Type":"ContainerDied","Data":"f7907928918c239aa4669ee8f9aa050a50436457c7e0181fa72319d611a6eae5"} Nov 03 02:16:54.141319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.140955 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" event={"ID":"88714959-afe9-4294-b124-fca1f9d0c4f3","Type":"ContainerDied","Data":"3335eabe0ec96999418f0dca7ef5858a9e8f9dc2a999d73c1f4fb316a1f6466c"} Nov 03 02:16:54.141319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.141033 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7" Nov 03 02:16:54.144932 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.144905 5463 generic.go:356] "Generic (PLEG): container finished" podID="2617013b-bed0-446d-a597-31bb323d1c6a" containerID="d47147148bca0b49acfcb877762bf35db07ee00784f8854099753e9fc5625abf" exitCode=0 Nov 03 02:16:54.145119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.144951 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" event={"ID":"2617013b-bed0-446d-a597-31bb323d1c6a","Type":"ContainerDied","Data":"d47147148bca0b49acfcb877762bf35db07ee00784f8854099753e9fc5625abf"} Nov 03 02:16:54.145119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.144993 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" event={"ID":"2617013b-bed0-446d-a597-31bb323d1c6a","Type":"ContainerDied","Data":"e4e4faffbe789e4b8104a7da712f774ebf3db094ddbc71a1c5c679350154dbe2"} Nov 03 02:16:54.147238 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.147218 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx" Nov 03 02:16:54.147496 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.147471 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-workload-certs\") pod \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " Nov 03 02:16:54.147573 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.147530 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istiod-ca-cert\") pod \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " Nov 03 02:16:54.147634 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.147593 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-credential-socket\") pod \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " Nov 03 02:16:54.149686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.147664 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-workload-socket\") pod \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " Nov 03 02:16:54.149686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.147783 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-workload-certs" pod "9ec04276-3f4d-4bdf-96bf-37f93fd0913e" (UID: "9ec04276-3f4d-4bdf-96bf-37f93fd0913e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:54.149686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.147848 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-kube-api-access-2t4vz\" (UniqueName: \"kubernetes.io/projected/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-kube-api-access-2t4vz\") pod \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " Nov 03 02:16:54.149686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.148164 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-credential-socket" pod "9ec04276-3f4d-4bdf-96bf-37f93fd0913e" (UID: "9ec04276-3f4d-4bdf-96bf-37f93fd0913e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:54.149686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.148169 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istiod-ca-cert" pod "9ec04276-3f4d-4bdf-96bf-37f93fd0913e" (UID: "9ec04276-3f4d-4bdf-96bf-37f93fd0913e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:16:54.149686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.148378 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-workload-socket" pod "9ec04276-3f4d-4bdf-96bf-37f93fd0913e" (UID: "9ec04276-3f4d-4bdf-96bf-37f93fd0913e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:54.149686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.148530 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-data\") pod \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " Nov 03 02:16:54.149686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.148589 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-envoy\") pod \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " Nov 03 02:16:54.149686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.148768 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-token\" (UniqueName: \"kubernetes.io/projected/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-token\") pod \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " Nov 03 02:16:54.149686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.148807 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-podinfo\") pod \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\" (UID: \"9ec04276-3f4d-4bdf-96bf-37f93fd0913e\") " Nov 03 02:16:54.150499 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.150475 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-data" pod "9ec04276-3f4d-4bdf-96bf-37f93fd0913e" (UID: "9ec04276-3f4d-4bdf-96bf-37f93fd0913e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:54.169622 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.169574 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-envoy" pod "9ec04276-3f4d-4bdf-96bf-37f93fd0913e" (UID: "9ec04276-3f4d-4bdf-96bf-37f93fd0913e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:54.170025 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.169964 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-kube-api-access-2t4vz" pod "9ec04276-3f4d-4bdf-96bf-37f93fd0913e" (UID: "9ec04276-3f4d-4bdf-96bf-37f93fd0913e"). InnerVolumeSpecName "kube-api-access-2t4vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:54.171745 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.171709 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-podinfo" pod "9ec04276-3f4d-4bdf-96bf-37f93fd0913e" (UID: "9ec04276-3f4d-4bdf-96bf-37f93fd0913e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:16:54.172042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.172018 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-token" pod "9ec04276-3f4d-4bdf-96bf-37f93fd0913e" (UID: "9ec04276-3f4d-4bdf-96bf-37f93fd0913e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:54.186968 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.185368 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:16:54.186968 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.186679 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:16:54.187860 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.187810 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:16:54.207683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.206913 5463 scope.go:122] "RemoveContainer" containerID="28fd5f6ba5091a131cb357b2f1635f94929e6ec12bcb32112d8adcff51847e90" Nov 03 02:16:54.207683 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:16:54.207348 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28fd5f6ba5091a131cb357b2f1635f94929e6ec12bcb32112d8adcff51847e90\": container with ID starting with 28fd5f6ba5091a131cb357b2f1635f94929e6ec12bcb32112d8adcff51847e90 not found: ID does not exist" containerID="28fd5f6ba5091a131cb357b2f1635f94929e6ec12bcb32112d8adcff51847e90" Nov 03 02:16:54.207683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.207385 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28fd5f6ba5091a131cb357b2f1635f94929e6ec12bcb32112d8adcff51847e90"} err="failed to get container status \"28fd5f6ba5091a131cb357b2f1635f94929e6ec12bcb32112d8adcff51847e90\": rpc error: code = NotFound desc = could not find container \"28fd5f6ba5091a131cb357b2f1635f94929e6ec12bcb32112d8adcff51847e90\": container with ID starting with 28fd5f6ba5091a131cb357b2f1635f94929e6ec12bcb32112d8adcff51847e90 not found: ID does not exist" Nov 03 02:16:54.207683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.207425 5463 scope.go:122] "RemoveContainer" containerID="8649bcda78befe6cb20fc947e18bfc41f77ed6a468eb1bd07e501eb688529f8d" Nov 03 02:16:54.225550 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.224372 5463 scope.go:122] "RemoveContainer" containerID="bdc92936e7c5552b0cdaa645cabf9cb7a0f50f330262a1963352820dfa82ae06" Nov 03 02:16:54.246696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.246677 5463 scope.go:122] "RemoveContainer" containerID="8649bcda78befe6cb20fc947e18bfc41f77ed6a468eb1bd07e501eb688529f8d" Nov 03 02:16:54.247005 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:16:54.246981 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8649bcda78befe6cb20fc947e18bfc41f77ed6a468eb1bd07e501eb688529f8d\": container with ID starting with 8649bcda78befe6cb20fc947e18bfc41f77ed6a468eb1bd07e501eb688529f8d not found: ID does not exist" containerID="8649bcda78befe6cb20fc947e18bfc41f77ed6a468eb1bd07e501eb688529f8d" Nov 03 02:16:54.247056 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.247016 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8649bcda78befe6cb20fc947e18bfc41f77ed6a468eb1bd07e501eb688529f8d"} err="failed to get container status \"8649bcda78befe6cb20fc947e18bfc41f77ed6a468eb1bd07e501eb688529f8d\": rpc error: code = NotFound desc = could not find container \"8649bcda78befe6cb20fc947e18bfc41f77ed6a468eb1bd07e501eb688529f8d\": container with ID starting with 8649bcda78befe6cb20fc947e18bfc41f77ed6a468eb1bd07e501eb688529f8d not found: ID does not exist" Nov 03 02:16:54.247056 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.247034 5463 scope.go:122] "RemoveContainer" containerID="bdc92936e7c5552b0cdaa645cabf9cb7a0f50f330262a1963352820dfa82ae06" Nov 03 02:16:54.247370 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:16:54.247349 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc92936e7c5552b0cdaa645cabf9cb7a0f50f330262a1963352820dfa82ae06\": container with ID starting with bdc92936e7c5552b0cdaa645cabf9cb7a0f50f330262a1963352820dfa82ae06 not found: ID does not exist" containerID="bdc92936e7c5552b0cdaa645cabf9cb7a0f50f330262a1963352820dfa82ae06" Nov 03 02:16:54.247438 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.247375 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc92936e7c5552b0cdaa645cabf9cb7a0f50f330262a1963352820dfa82ae06"} err="failed to get container status \"bdc92936e7c5552b0cdaa645cabf9cb7a0f50f330262a1963352820dfa82ae06\": rpc error: code = NotFound desc = could not find container \"bdc92936e7c5552b0cdaa645cabf9cb7a0f50f330262a1963352820dfa82ae06\": container with ID starting with bdc92936e7c5552b0cdaa645cabf9cb7a0f50f330262a1963352820dfa82ae06 not found: ID does not exist" Nov 03 02:16:54.247438 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.247392 5463 scope.go:122] "RemoveContainer" containerID="01c1dd81d2ac1849ef17d16215b7b100b407a0698d4d277555af1db42e47d994" Nov 03 02:16:54.250850 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.250683 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.250850 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.250709 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.250850 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.250726 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.250850 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.250744 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.250850 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.250760 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.250850 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.250779 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.250850 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.250795 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t4vz\" (UniqueName: \"kubernetes.io/projected/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-kube-api-access-2t4vz\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.250850 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.250812 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.250850 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.250827 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec04276-3f4d-4bdf-96bf-37f93fd0913e-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.261236 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.261167 5463 scope.go:122] "RemoveContainer" containerID="01c1dd81d2ac1849ef17d16215b7b100b407a0698d4d277555af1db42e47d994" Nov 03 02:16:54.261509 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:16:54.261483 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c1dd81d2ac1849ef17d16215b7b100b407a0698d4d277555af1db42e47d994\": container with ID starting with 01c1dd81d2ac1849ef17d16215b7b100b407a0698d4d277555af1db42e47d994 not found: ID does not exist" containerID="01c1dd81d2ac1849ef17d16215b7b100b407a0698d4d277555af1db42e47d994" Nov 03 02:16:54.261575 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.261516 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c1dd81d2ac1849ef17d16215b7b100b407a0698d4d277555af1db42e47d994"} err="failed to get container status \"01c1dd81d2ac1849ef17d16215b7b100b407a0698d4d277555af1db42e47d994\": rpc error: code = NotFound desc = could not find container \"01c1dd81d2ac1849ef17d16215b7b100b407a0698d4d277555af1db42e47d994\": container with ID starting with 01c1dd81d2ac1849ef17d16215b7b100b407a0698d4d277555af1db42e47d994 not found: ID does not exist" Nov 03 02:16:54.261575 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.261531 5463 scope.go:122] "RemoveContainer" containerID="f7907928918c239aa4669ee8f9aa050a50436457c7e0181fa72319d611a6eae5" Nov 03 02:16:54.273451 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.273432 5463 scope.go:122] "RemoveContainer" containerID="f7907928918c239aa4669ee8f9aa050a50436457c7e0181fa72319d611a6eae5" Nov 03 02:16:54.273714 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:16:54.273698 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7907928918c239aa4669ee8f9aa050a50436457c7e0181fa72319d611a6eae5\": container with ID starting with f7907928918c239aa4669ee8f9aa050a50436457c7e0181fa72319d611a6eae5 not found: ID does not exist" containerID="f7907928918c239aa4669ee8f9aa050a50436457c7e0181fa72319d611a6eae5" Nov 03 02:16:54.273782 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.273723 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7907928918c239aa4669ee8f9aa050a50436457c7e0181fa72319d611a6eae5"} err="failed to get container status \"f7907928918c239aa4669ee8f9aa050a50436457c7e0181fa72319d611a6eae5\": rpc error: code = NotFound desc = could not find container \"f7907928918c239aa4669ee8f9aa050a50436457c7e0181fa72319d611a6eae5\": container with ID starting with f7907928918c239aa4669ee8f9aa050a50436457c7e0181fa72319d611a6eae5 not found: ID does not exist" Nov 03 02:16:54.273782 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.273743 5463 scope.go:122] "RemoveContainer" containerID="d47147148bca0b49acfcb877762bf35db07ee00784f8854099753e9fc5625abf" Nov 03 02:16:54.283177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.283153 5463 scope.go:122] "RemoveContainer" containerID="d47147148bca0b49acfcb877762bf35db07ee00784f8854099753e9fc5625abf" Nov 03 02:16:54.283517 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:16:54.283499 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47147148bca0b49acfcb877762bf35db07ee00784f8854099753e9fc5625abf\": container with ID starting with d47147148bca0b49acfcb877762bf35db07ee00784f8854099753e9fc5625abf not found: ID does not exist" containerID="d47147148bca0b49acfcb877762bf35db07ee00784f8854099753e9fc5625abf" Nov 03 02:16:54.283596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.283522 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47147148bca0b49acfcb877762bf35db07ee00784f8854099753e9fc5625abf"} err="failed to get container status \"d47147148bca0b49acfcb877762bf35db07ee00784f8854099753e9fc5625abf\": rpc error: code = NotFound desc = could not find container \"d47147148bca0b49acfcb877762bf35db07ee00784f8854099753e9fc5625abf\": container with ID starting with d47147148bca0b49acfcb877762bf35db07ee00784f8854099753e9fc5625abf not found: ID does not exist" Nov 03 02:16:54.317238 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.317209 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:54.409202 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.409157 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5"] Nov 03 02:16:54.412966 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.412940 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-xz6r5"] Nov 03 02:16:54.469424 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.467724 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="731a025d-7302-4a22-8b43-70f96d8f05bb" path="/var/lib/kubelet/pods/731a025d-7302-4a22-8b43-70f96d8f05bb/volumes" Nov 03 02:16:54.620109 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.619859 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 02:16:54.620109 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.619917 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" podUID="13e969fe-f189-40f2-b929-e462f6d4b86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 02:16:54.652770 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.652715 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7"] Nov 03 02:16:54.659639 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.659543 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-97kz7"] Nov 03 02:16:54.665234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.665200 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd"] Nov 03 02:16:54.782434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.782401 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f"] Nov 03 02:16:54.795236 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.795118 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:16:54.815924 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.815896 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx"] Nov 03 02:16:54.820688 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.820667 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-pdlbx"] Nov 03 02:16:54.845644 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.845620 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:16:54.864727 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.864696 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2746f922-b469-4383-90bb-3be08f9256d7-istio-token\" (UniqueName: \"kubernetes.io/projected/2746f922-b469-4383-90bb-3be08f9256d7-istio-token\") pod \"2746f922-b469-4383-90bb-3be08f9256d7\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " Nov 03 02:16:54.864966 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.864946 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2746f922-b469-4383-90bb-3be08f9256d7-kube-api-access-w7vz9\" (UniqueName: \"kubernetes.io/projected/2746f922-b469-4383-90bb-3be08f9256d7-kube-api-access-w7vz9\") pod \"2746f922-b469-4383-90bb-3be08f9256d7\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " Nov 03 02:16:54.865179 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.865159 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-workload-socket\") pod \"2746f922-b469-4383-90bb-3be08f9256d7\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " Nov 03 02:16:54.865413 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.865397 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-istio-data\") pod \"2746f922-b469-4383-90bb-3be08f9256d7\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " Nov 03 02:16:54.865524 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.865510 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-credential-socket\") pod \"2746f922-b469-4383-90bb-3be08f9256d7\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " Nov 03 02:16:54.865612 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.865586 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-workload-socket" pod "2746f922-b469-4383-90bb-3be08f9256d7" (UID: "2746f922-b469-4383-90bb-3be08f9256d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:54.865709 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.865696 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-workload-certs\") pod \"2746f922-b469-4383-90bb-3be08f9256d7\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " Nov 03 02:16:54.866792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.865699 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-istio-data" pod "2746f922-b469-4383-90bb-3be08f9256d7" (UID: "2746f922-b469-4383-90bb-3be08f9256d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:54.866792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.865867 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2746f922-b469-4383-90bb-3be08f9256d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2746f922-b469-4383-90bb-3be08f9256d7-istio-podinfo\") pod \"2746f922-b469-4383-90bb-3be08f9256d7\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " Nov 03 02:16:54.866792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.865952 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-istio-envoy\") pod \"2746f922-b469-4383-90bb-3be08f9256d7\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " Nov 03 02:16:54.866792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.866051 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2746f922-b469-4383-90bb-3be08f9256d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2746f922-b469-4383-90bb-3be08f9256d7-istiod-ca-cert\") pod \"2746f922-b469-4383-90bb-3be08f9256d7\" (UID: \"2746f922-b469-4383-90bb-3be08f9256d7\") " Nov 03 02:16:54.867725 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.866360 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-credential-socket" pod "2746f922-b469-4383-90bb-3be08f9256d7" (UID: "2746f922-b469-4383-90bb-3be08f9256d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:54.867832 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.866654 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2746f922-b469-4383-90bb-3be08f9256d7-istiod-ca-cert" pod "2746f922-b469-4383-90bb-3be08f9256d7" (UID: "2746f922-b469-4383-90bb-3be08f9256d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:16:54.867832 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.866772 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-workload-certs" pod "2746f922-b469-4383-90bb-3be08f9256d7" (UID: "2746f922-b469-4383-90bb-3be08f9256d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:54.867832 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.866705 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.867832 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.867777 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.870566 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.870530 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2746f922-b469-4383-90bb-3be08f9256d7-istio-podinfo" pod "2746f922-b469-4383-90bb-3be08f9256d7" (UID: "2746f922-b469-4383-90bb-3be08f9256d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:16:54.872313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.872262 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2746f922-b469-4383-90bb-3be08f9256d7-istio-token" pod "2746f922-b469-4383-90bb-3be08f9256d7" (UID: "2746f922-b469-4383-90bb-3be08f9256d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:54.872585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.872557 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-istio-envoy" pod "2746f922-b469-4383-90bb-3be08f9256d7" (UID: "2746f922-b469-4383-90bb-3be08f9256d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:54.878802 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.878778 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2746f922-b469-4383-90bb-3be08f9256d7-kube-api-access-w7vz9" pod "2746f922-b469-4383-90bb-3be08f9256d7" (UID: "2746f922-b469-4383-90bb-3be08f9256d7"). InnerVolumeSpecName "kube-api-access-w7vz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:54.901719 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.901692 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:16:54.932139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.932065 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf"] Nov 03 02:16:54.938384 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.938361 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-bxhzf"] Nov 03 02:16:54.968987 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.968953 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-envoy\") pod \"13e969fe-f189-40f2-b929-e462f6d4b86b\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " Nov 03 02:16:54.969140 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.969038 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istiod-ca-cert\") pod \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " Nov 03 02:16:54.969140 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.969080 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-workload-socket\") pod \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " Nov 03 02:16:54.969140 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.969107 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-envoy\") pod \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " Nov 03 02:16:54.969375 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.969141 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-workload-certs\") pod \"13e969fe-f189-40f2-b929-e462f6d4b86b\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " Nov 03 02:16:54.969375 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.969257 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-credential-socket\") pod \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " Nov 03 02:16:54.969375 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.969300 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-token\" (UniqueName: \"kubernetes.io/projected/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-token\") pod \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " Nov 03 02:16:54.969375 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.969338 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-workload-certs\") pod \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " Nov 03 02:16:54.969640 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.969376 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-data\") pod \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " Nov 03 02:16:54.969640 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.969414 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-podinfo\") pod \"13e969fe-f189-40f2-b929-e462f6d4b86b\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " Nov 03 02:16:54.969640 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.969443 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-podinfo\") pod \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " Nov 03 02:16:54.969640 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.969486 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e969fe-f189-40f2-b929-e462f6d4b86b-kube-api-access-fb96n\" (UniqueName: \"kubernetes.io/projected/13e969fe-f189-40f2-b929-e462f6d4b86b-kube-api-access-fb96n\") pod \"13e969fe-f189-40f2-b929-e462f6d4b86b\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " Nov 03 02:16:54.970215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.969664 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13e969fe-f189-40f2-b929-e462f6d4b86b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e969fe-f189-40f2-b929-e462f6d4b86b-istiod-ca-cert\") pod \"13e969fe-f189-40f2-b929-e462f6d4b86b\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " Nov 03 02:16:54.970215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.969733 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-workload-socket\") pod \"13e969fe-f189-40f2-b929-e462f6d4b86b\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " Nov 03 02:16:54.970215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.969768 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-credential-socket\") pod \"13e969fe-f189-40f2-b929-e462f6d4b86b\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " Nov 03 02:16:54.970215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.969822 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-data\") pod \"13e969fe-f189-40f2-b929-e462f6d4b86b\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " Nov 03 02:16:54.970215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.969858 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-token\" (UniqueName: \"kubernetes.io/projected/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-token\") pod \"13e969fe-f189-40f2-b929-e462f6d4b86b\" (UID: \"13e969fe-f189-40f2-b929-e462f6d4b86b\") " Nov 03 02:16:54.970215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.969929 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc2f2fb4-5a6f-4778-90e5-9a857204f663-kube-api-access-rck85\" (UniqueName: \"kubernetes.io/projected/cc2f2fb4-5a6f-4778-90e5-9a857204f663-kube-api-access-rck85\") pod \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\" (UID: \"cc2f2fb4-5a6f-4778-90e5-9a857204f663\") " Nov 03 02:16:54.970799 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.970778 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2746f922-b469-4383-90bb-3be08f9256d7-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.971310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.970803 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7vz9\" (UniqueName: \"kubernetes.io/projected/2746f922-b469-4383-90bb-3be08f9256d7-kube-api-access-w7vz9\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.971310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.970818 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.971310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.970831 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.971310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.970844 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2746f922-b469-4383-90bb-3be08f9256d7-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.971310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.970858 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2746f922-b469-4383-90bb-3be08f9256d7-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:54.971310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:54.970871 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2746f922-b469-4383-90bb-3be08f9256d7-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.000721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.000320 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-envoy" pod "13e969fe-f189-40f2-b929-e462f6d4b86b" (UID: "13e969fe-f189-40f2-b929-e462f6d4b86b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:55.001964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.001021 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istiod-ca-cert" pod "cc2f2fb4-5a6f-4778-90e5-9a857204f663" (UID: "cc2f2fb4-5a6f-4778-90e5-9a857204f663"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:16:55.002484 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.002169 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-workload-socket" pod "cc2f2fb4-5a6f-4778-90e5-9a857204f663" (UID: "cc2f2fb4-5a6f-4778-90e5-9a857204f663"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:55.011279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.008920 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-envoy" pod "cc2f2fb4-5a6f-4778-90e5-9a857204f663" (UID: "cc2f2fb4-5a6f-4778-90e5-9a857204f663"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:55.011279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.009147 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-workload-certs" pod "13e969fe-f189-40f2-b929-e462f6d4b86b" (UID: "13e969fe-f189-40f2-b929-e462f6d4b86b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:55.011279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.009372 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-credential-socket" pod "cc2f2fb4-5a6f-4778-90e5-9a857204f663" (UID: "cc2f2fb4-5a6f-4778-90e5-9a857204f663"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:55.011279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.011103 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-token" pod "cc2f2fb4-5a6f-4778-90e5-9a857204f663" (UID: "cc2f2fb4-5a6f-4778-90e5-9a857204f663"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:55.011535 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.011290 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-workload-certs" pod "cc2f2fb4-5a6f-4778-90e5-9a857204f663" (UID: "cc2f2fb4-5a6f-4778-90e5-9a857204f663"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:55.011535 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.011441 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-data" pod "cc2f2fb4-5a6f-4778-90e5-9a857204f663" (UID: "cc2f2fb4-5a6f-4778-90e5-9a857204f663"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:55.013264 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.013013 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-podinfo" pod "13e969fe-f189-40f2-b929-e462f6d4b86b" (UID: "13e969fe-f189-40f2-b929-e462f6d4b86b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:16:55.014411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.014035 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-workload-socket" pod "13e969fe-f189-40f2-b929-e462f6d4b86b" (UID: "13e969fe-f189-40f2-b929-e462f6d4b86b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:55.014411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.014326 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e969fe-f189-40f2-b929-e462f6d4b86b-istiod-ca-cert" pod "13e969fe-f189-40f2-b929-e462f6d4b86b" (UID: "13e969fe-f189-40f2-b929-e462f6d4b86b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:16:55.014738 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.014619 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-data" pod "13e969fe-f189-40f2-b929-e462f6d4b86b" (UID: "13e969fe-f189-40f2-b929-e462f6d4b86b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:55.014738 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.014637 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-credential-socket" pod "13e969fe-f189-40f2-b929-e462f6d4b86b" (UID: "13e969fe-f189-40f2-b929-e462f6d4b86b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:55.015360 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.015145 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-podinfo" pod "cc2f2fb4-5a6f-4778-90e5-9a857204f663" (UID: "cc2f2fb4-5a6f-4778-90e5-9a857204f663"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:16:55.015360 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.015325 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2f2fb4-5a6f-4778-90e5-9a857204f663-kube-api-access-rck85" pod "cc2f2fb4-5a6f-4778-90e5-9a857204f663" (UID: "cc2f2fb4-5a6f-4778-90e5-9a857204f663"). InnerVolumeSpecName "kube-api-access-rck85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:55.017310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.016829 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e969fe-f189-40f2-b929-e462f6d4b86b-kube-api-access-fb96n" pod "13e969fe-f189-40f2-b929-e462f6d4b86b" (UID: "13e969fe-f189-40f2-b929-e462f6d4b86b"). InnerVolumeSpecName "kube-api-access-fb96n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:55.023229 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.022070 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-token" pod "13e969fe-f189-40f2-b929-e462f6d4b86b" (UID: "13e969fe-f189-40f2-b929-e462f6d4b86b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:55.056299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.056263 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:55.073954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.073910 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.073954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.073955 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.074154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.073971 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.074154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.073996 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.074154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.074023 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb96n\" (UniqueName: \"kubernetes.io/projected/13e969fe-f189-40f2-b929-e462f6d4b86b-kube-api-access-fb96n\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.074154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.074039 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e969fe-f189-40f2-b929-e462f6d4b86b-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.074154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.074052 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.074154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.074066 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.074154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.074091 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.074154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.074105 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.074154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.074119 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rck85\" (UniqueName: \"kubernetes.io/projected/cc2f2fb4-5a6f-4778-90e5-9a857204f663-kube-api-access-rck85\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.074154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.074133 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.074154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.074147 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.075014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.074173 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.075014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.074205 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.075014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.074220 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e969fe-f189-40f2-b929-e462f6d4b86b-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.075014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.074233 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2f2fb4-5a6f-4778-90e5-9a857204f663-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.075014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.074246 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2f2fb4-5a6f-4778-90e5-9a857204f663-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.163137 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.163098 5463 generic.go:356] "Generic (PLEG): container finished" podID="13e969fe-f189-40f2-b929-e462f6d4b86b" containerID="141f045aa4d400f57e8e3bf8e8ce8f30fe30ae85ee65e7fce4e42303090a2003" exitCode=0 Nov 03 02:16:55.163502 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.163476 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" event={"ID":"13e969fe-f189-40f2-b929-e462f6d4b86b","Type":"ContainerDied","Data":"141f045aa4d400f57e8e3bf8e8ce8f30fe30ae85ee65e7fce4e42303090a2003"} Nov 03 02:16:55.163712 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.163696 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" Nov 03 02:16:55.167888 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.167856 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7" event={"ID":"13e969fe-f189-40f2-b929-e462f6d4b86b","Type":"ContainerDied","Data":"59af71994020b2bed3ec77c86d1dbe46dba12fc875e0bf46ad58313a73c51184"} Nov 03 02:16:55.168003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.167914 5463 scope.go:122] "RemoveContainer" containerID="141f045aa4d400f57e8e3bf8e8ce8f30fe30ae85ee65e7fce4e42303090a2003" Nov 03 02:16:55.171909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.171532 5463 generic.go:356] "Generic (PLEG): container finished" podID="cc2f2fb4-5a6f-4778-90e5-9a857204f663" containerID="35464c1f7916006f4cedd3f0b5ba10a46959f8a5e4b006e1097fe0ee2a577f0e" exitCode=0 Nov 03 02:16:55.171909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.171620 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" event={"ID":"cc2f2fb4-5a6f-4778-90e5-9a857204f663","Type":"ContainerDied","Data":"35464c1f7916006f4cedd3f0b5ba10a46959f8a5e4b006e1097fe0ee2a577f0e"} Nov 03 02:16:55.171909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.171663 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" event={"ID":"cc2f2fb4-5a6f-4778-90e5-9a857204f663","Type":"ContainerDied","Data":"844dcac3ae89c7bcd82145c75d87a37bac9c537365e622c26a8404a0533267c6"} Nov 03 02:16:55.171909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.171759 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8" Nov 03 02:16:55.182990 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.182701 5463 generic.go:356] "Generic (PLEG): container finished" podID="32ef5394-19c0-4b33-84f6-2087dede03a0" containerID="0cfba01290b039c4902ed6b049e39f9411f18fc8eb7fa24bb60395651e1e8792" exitCode=0 Nov 03 02:16:55.183185 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.182745 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" event={"ID":"32ef5394-19c0-4b33-84f6-2087dede03a0","Type":"ContainerDied","Data":"0cfba01290b039c4902ed6b049e39f9411f18fc8eb7fa24bb60395651e1e8792"} Nov 03 02:16:55.187504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.187481 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" event={"ID":"32ef5394-19c0-4b33-84f6-2087dede03a0","Type":"ContainerDied","Data":"0d14eb679cb15cfdcfb36634c6db2c3fd9a00320ab2e289d43c697775542d5db"} Nov 03 02:16:55.187642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.187626 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d14eb679cb15cfdcfb36634c6db2c3fd9a00320ab2e289d43c697775542d5db" Nov 03 02:16:55.383276 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.306855 5463 scope.go:122] "RemoveContainer" containerID="141f045aa4d400f57e8e3bf8e8ce8f30fe30ae85ee65e7fce4e42303090a2003" Nov 03 02:16:55.383276 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.323940 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:16:55.385703 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:16:55.385184 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141f045aa4d400f57e8e3bf8e8ce8f30fe30ae85ee65e7fce4e42303090a2003\": container with ID starting with 141f045aa4d400f57e8e3bf8e8ce8f30fe30ae85ee65e7fce4e42303090a2003 not found: ID does not exist" containerID="141f045aa4d400f57e8e3bf8e8ce8f30fe30ae85ee65e7fce4e42303090a2003" Nov 03 02:16:55.385703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.385269 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141f045aa4d400f57e8e3bf8e8ce8f30fe30ae85ee65e7fce4e42303090a2003"} err="failed to get container status \"141f045aa4d400f57e8e3bf8e8ce8f30fe30ae85ee65e7fce4e42303090a2003\": rpc error: code = NotFound desc = could not find container \"141f045aa4d400f57e8e3bf8e8ce8f30fe30ae85ee65e7fce4e42303090a2003\": container with ID starting with 141f045aa4d400f57e8e3bf8e8ce8f30fe30ae85ee65e7fce4e42303090a2003 not found: ID does not exist" Nov 03 02:16:55.385703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.385293 5463 scope.go:122] "RemoveContainer" containerID="35464c1f7916006f4cedd3f0b5ba10a46959f8a5e4b006e1097fe0ee2a577f0e" Nov 03 02:16:55.385703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.385537 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:55.386654 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.386605 5463 generic.go:356] "Generic (PLEG): container finished" podID="2746f922-b469-4383-90bb-3be08f9256d7" containerID="cb15eb4b66f3990c6f574a0ab10364eee39b9a466f6aeaa245dbdd7e1dc8942c" exitCode=0 Nov 03 02:16:55.391572 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.388313 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" Nov 03 02:16:55.393506 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.393403 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" event={"ID":"2746f922-b469-4383-90bb-3be08f9256d7","Type":"ContainerDied","Data":"cb15eb4b66f3990c6f574a0ab10364eee39b9a466f6aeaa245dbdd7e1dc8942c"} Nov 03 02:16:55.393506 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.393462 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p" event={"ID":"2746f922-b469-4383-90bb-3be08f9256d7","Type":"ContainerDied","Data":"a5fff26fd56eba8e04ad5a9c5193cbf9ac045c2928ae6acc00fee7ddf457ef37"} Nov 03 02:16:55.424810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.424788 5463 scope.go:122] "RemoveContainer" containerID="35464c1f7916006f4cedd3f0b5ba10a46959f8a5e4b006e1097fe0ee2a577f0e" Nov 03 02:16:55.425168 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:16:55.425145 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35464c1f7916006f4cedd3f0b5ba10a46959f8a5e4b006e1097fe0ee2a577f0e\": container with ID starting with 35464c1f7916006f4cedd3f0b5ba10a46959f8a5e4b006e1097fe0ee2a577f0e not found: ID does not exist" containerID="35464c1f7916006f4cedd3f0b5ba10a46959f8a5e4b006e1097fe0ee2a577f0e" Nov 03 02:16:55.425289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.425177 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35464c1f7916006f4cedd3f0b5ba10a46959f8a5e4b006e1097fe0ee2a577f0e"} err="failed to get container status \"35464c1f7916006f4cedd3f0b5ba10a46959f8a5e4b006e1097fe0ee2a577f0e\": rpc error: code = NotFound desc = could not find container \"35464c1f7916006f4cedd3f0b5ba10a46959f8a5e4b006e1097fe0ee2a577f0e\": container with ID starting with 35464c1f7916006f4cedd3f0b5ba10a46959f8a5e4b006e1097fe0ee2a577f0e not found: ID does not exist" Nov 03 02:16:55.425289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.425213 5463 scope.go:122] "RemoveContainer" containerID="cb15eb4b66f3990c6f574a0ab10364eee39b9a466f6aeaa245dbdd7e1dc8942c" Nov 03 02:16:55.431383 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:16:55.431352 5463 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e969fe_f189_40f2_b929_e462f6d4b86b.slice/crio-59af71994020b2bed3ec77c86d1dbe46dba12fc875e0bf46ad58313a73c51184\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e969fe_f189_40f2_b929_e462f6d4b86b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2f2fb4_5a6f_4778_90e5_9a857204f663.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2f2fb4_5a6f_4778_90e5_9a857204f663.slice/crio-844dcac3ae89c7bcd82145c75d87a37bac9c537365e622c26a8404a0533267c6\": RecentStats: unable to find data in memory cache]" Nov 03 02:16:55.436832 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.436810 5463 scope.go:122] "RemoveContainer" containerID="4665c8b21421aec75f704c6ad2b2457e3cf9c6bfd07b05cf5a00877084b195e2" Nov 03 02:16:55.471954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.471917 5463 scope.go:122] "RemoveContainer" containerID="cb15eb4b66f3990c6f574a0ab10364eee39b9a466f6aeaa245dbdd7e1dc8942c" Nov 03 02:16:55.472337 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:16:55.472313 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb15eb4b66f3990c6f574a0ab10364eee39b9a466f6aeaa245dbdd7e1dc8942c\": container with ID starting with cb15eb4b66f3990c6f574a0ab10364eee39b9a466f6aeaa245dbdd7e1dc8942c not found: ID does not exist" containerID="cb15eb4b66f3990c6f574a0ab10364eee39b9a466f6aeaa245dbdd7e1dc8942c" Nov 03 02:16:55.472337 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.472347 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb15eb4b66f3990c6f574a0ab10364eee39b9a466f6aeaa245dbdd7e1dc8942c"} err="failed to get container status \"cb15eb4b66f3990c6f574a0ab10364eee39b9a466f6aeaa245dbdd7e1dc8942c\": rpc error: code = NotFound desc = could not find container \"cb15eb4b66f3990c6f574a0ab10364eee39b9a466f6aeaa245dbdd7e1dc8942c\": container with ID starting with cb15eb4b66f3990c6f574a0ab10364eee39b9a466f6aeaa245dbdd7e1dc8942c not found: ID does not exist" Nov 03 02:16:55.472614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.472372 5463 scope.go:122] "RemoveContainer" containerID="4665c8b21421aec75f704c6ad2b2457e3cf9c6bfd07b05cf5a00877084b195e2" Nov 03 02:16:55.472690 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:16:55.472661 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4665c8b21421aec75f704c6ad2b2457e3cf9c6bfd07b05cf5a00877084b195e2\": container with ID starting with 4665c8b21421aec75f704c6ad2b2457e3cf9c6bfd07b05cf5a00877084b195e2 not found: ID does not exist" containerID="4665c8b21421aec75f704c6ad2b2457e3cf9c6bfd07b05cf5a00877084b195e2" Nov 03 02:16:55.472759 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.472691 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4665c8b21421aec75f704c6ad2b2457e3cf9c6bfd07b05cf5a00877084b195e2"} err="failed to get container status \"4665c8b21421aec75f704c6ad2b2457e3cf9c6bfd07b05cf5a00877084b195e2\": rpc error: code = NotFound desc = could not find container \"4665c8b21421aec75f704c6ad2b2457e3cf9c6bfd07b05cf5a00877084b195e2\": container with ID starting with 4665c8b21421aec75f704c6ad2b2457e3cf9c6bfd07b05cf5a00877084b195e2 not found: ID does not exist" Nov 03 02:16:55.492316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.487615 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ef5394-19c0-4b33-84f6-2087dede03a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ef5394-19c0-4b33-84f6-2087dede03a0-istiod-ca-cert\") pod \"32ef5394-19c0-4b33-84f6-2087dede03a0\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " Nov 03 02:16:55.492316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.487665 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ef5394-19c0-4b33-84f6-2087dede03a0-kube-api-access-lqvs9\" (UniqueName: \"kubernetes.io/projected/32ef5394-19c0-4b33-84f6-2087dede03a0-kube-api-access-lqvs9\") pod \"32ef5394-19c0-4b33-84f6-2087dede03a0\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " Nov 03 02:16:55.492316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.487781 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-token\" (UniqueName: \"kubernetes.io/projected/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-token\") pod \"32ef5394-19c0-4b33-84f6-2087dede03a0\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " Nov 03 02:16:55.492316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.487814 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-podinfo\") pod \"32ef5394-19c0-4b33-84f6-2087dede03a0\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " Nov 03 02:16:55.492316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.487938 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-envoy\") pod \"32ef5394-19c0-4b33-84f6-2087dede03a0\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " Nov 03 02:16:55.492316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.487970 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-workload-socket\") pod \"32ef5394-19c0-4b33-84f6-2087dede03a0\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " Nov 03 02:16:55.492316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.488052 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-credential-socket\") pod \"32ef5394-19c0-4b33-84f6-2087dede03a0\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " Nov 03 02:16:55.492316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.488085 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-workload-certs\") pod \"32ef5394-19c0-4b33-84f6-2087dede03a0\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " Nov 03 02:16:55.492316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.488146 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-data\") pod \"32ef5394-19c0-4b33-84f6-2087dede03a0\" (UID: \"32ef5394-19c0-4b33-84f6-2087dede03a0\") " Nov 03 02:16:55.492316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.489477 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ef5394-19c0-4b33-84f6-2087dede03a0-istiod-ca-cert" pod "32ef5394-19c0-4b33-84f6-2087dede03a0" (UID: "32ef5394-19c0-4b33-84f6-2087dede03a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:16:55.492316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.490626 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-workload-certs" pod "32ef5394-19c0-4b33-84f6-2087dede03a0" (UID: "32ef5394-19c0-4b33-84f6-2087dede03a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:55.492316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.490868 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-data" pod "32ef5394-19c0-4b33-84f6-2087dede03a0" (UID: "32ef5394-19c0-4b33-84f6-2087dede03a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:55.492316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.492181 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-credential-socket" pod "32ef5394-19c0-4b33-84f6-2087dede03a0" (UID: "32ef5394-19c0-4b33-84f6-2087dede03a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:55.492959 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.492483 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-workload-socket" pod "32ef5394-19c0-4b33-84f6-2087dede03a0" (UID: "32ef5394-19c0-4b33-84f6-2087dede03a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:55.499343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.497142 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-podinfo" pod "32ef5394-19c0-4b33-84f6-2087dede03a0" (UID: "32ef5394-19c0-4b33-84f6-2087dede03a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:16:55.499343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.499156 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-token" pod "32ef5394-19c0-4b33-84f6-2087dede03a0" (UID: "32ef5394-19c0-4b33-84f6-2087dede03a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:55.499343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.499179 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-envoy" pod "32ef5394-19c0-4b33-84f6-2087dede03a0" (UID: "32ef5394-19c0-4b33-84f6-2087dede03a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:55.499343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.499252 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ef5394-19c0-4b33-84f6-2087dede03a0-kube-api-access-lqvs9" pod "32ef5394-19c0-4b33-84f6-2087dede03a0" (UID: "32ef5394-19c0-4b33-84f6-2087dede03a0"). InnerVolumeSpecName "kube-api-access-lqvs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:55.590523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.590353 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.590523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.590394 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.590523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.590413 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.590523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.590427 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.590523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.590440 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.590523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.590453 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ef5394-19c0-4b33-84f6-2087dede03a0-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.590523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.590468 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqvs9\" (UniqueName: \"kubernetes.io/projected/32ef5394-19c0-4b33-84f6-2087dede03a0-kube-api-access-lqvs9\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.590523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.590482 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.590523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.590495 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ef5394-19c0-4b33-84f6-2087dede03a0-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:55.684287 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.684249 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n"] Nov 03 02:16:55.687672 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.687617 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7f332d-openshift-default-fd7c999d7-vdv8n"] Nov 03 02:16:55.980702 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:55.980625 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:56.028701 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:56.028673 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8"] Nov 03 02:16:56.032077 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:56.032056 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-7tcd8"] Nov 03 02:16:56.141742 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:56.141709 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7"] Nov 03 02:16:56.145313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:56.145287 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9fts7"] Nov 03 02:16:56.281326 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:56.281235 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p"] Nov 03 02:16:56.291095 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:56.291063 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-pcn4p"] Nov 03 02:16:56.399040 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:56.399015 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq" Nov 03 02:16:56.456944 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:56.456908 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13e969fe-f189-40f2-b929-e462f6d4b86b" path="/var/lib/kubelet/pods/13e969fe-f189-40f2-b929-e462f6d4b86b/volumes" Nov 03 02:16:56.457629 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:56.457609 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2617013b-bed0-446d-a597-31bb323d1c6a" path="/var/lib/kubelet/pods/2617013b-bed0-446d-a597-31bb323d1c6a/volumes" Nov 03 02:16:56.458239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:56.458222 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2746f922-b469-4383-90bb-3be08f9256d7" path="/var/lib/kubelet/pods/2746f922-b469-4383-90bb-3be08f9256d7/volumes" Nov 03 02:16:56.459077 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:56.459062 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88714959-afe9-4294-b124-fca1f9d0c4f3" path="/var/lib/kubelet/pods/88714959-afe9-4294-b124-fca1f9d0c4f3/volumes" Nov 03 02:16:56.459530 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:56.459517 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec04276-3f4d-4bdf-96bf-37f93fd0913e" path="/var/lib/kubelet/pods/9ec04276-3f4d-4bdf-96bf-37f93fd0913e/volumes" Nov 03 02:16:56.459953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:56.459941 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc2f2fb4-5a6f-4778-90e5-9a857204f663" path="/var/lib/kubelet/pods/cc2f2fb4-5a6f-4778-90e5-9a857204f663/volumes" Nov 03 02:16:56.460696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:56.460671 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" path="/var/lib/kubelet/pods/dfeb7b50-ad6f-4fa2-b672-cb47242deb1e/volumes" Nov 03 02:16:56.567358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:56.567263 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq"] Nov 03 02:16:56.571244 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:56.571161 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-dwxdq"] Nov 03 02:16:58.445219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:58.445170 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ef5394-19c0-4b33-84f6-2087dede03a0" path="/var/lib/kubelet/pods/32ef5394-19c0-4b33-84f6-2087dede03a0/volumes" Nov 03 02:16:59.356488 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:59.356458 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:59.505517 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:59.505485 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:59.655611 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:59.655579 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:59.665988 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:59.665952 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd" podUID="5472d731-9bca-404c-a5a4-97370f5b04f7" containerName="payload-processing" containerID="cri-o://6a3566f3b9a32c7cbc95e4991cb48a2dcdd961eb4805358de0414caa001f9274" gracePeriod=25 Nov 03 02:16:59.782813 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:59.782761 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f" podUID="557af6d4-ae3b-4b48-ae87-75095fe70ec2" containerName="payload-pre-processing" containerID="cri-o://0e58fda70f3435c6a411bdfc3efc41add67e150892052ba85a5c1de17af938cb" gracePeriod=25 Nov 03 02:16:59.960144 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:16:59.960117 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd" Nov 03 02:17:00.066801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.066774 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5472d731-9bca-404c-a5a4-97370f5b04f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5472d731-9bca-404c-a5a4-97370f5b04f7-plugins-config-volume\") pod \"5472d731-9bca-404c-a5a4-97370f5b04f7\" (UID: \"5472d731-9bca-404c-a5a4-97370f5b04f7\") " Nov 03 02:17:00.066971 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.066957 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5472d731-9bca-404c-a5a4-97370f5b04f7-kube-api-access-56qm6\" (UniqueName: \"kubernetes.io/projected/5472d731-9bca-404c-a5a4-97370f5b04f7-kube-api-access-56qm6\") pod \"5472d731-9bca-404c-a5a4-97370f5b04f7\" (UID: \"5472d731-9bca-404c-a5a4-97370f5b04f7\") " Nov 03 02:17:00.067096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.067076 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5472d731-9bca-404c-a5a4-97370f5b04f7-plugins-config-volume" pod "5472d731-9bca-404c-a5a4-97370f5b04f7" (UID: "5472d731-9bca-404c-a5a4-97370f5b04f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:17:00.067375 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.067359 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5472d731-9bca-404c-a5a4-97370f5b04f7-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:00.068244 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.068180 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f" Nov 03 02:17:00.068906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.068891 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5472d731-9bca-404c-a5a4-97370f5b04f7-kube-api-access-56qm6" pod "5472d731-9bca-404c-a5a4-97370f5b04f7" (UID: "5472d731-9bca-404c-a5a4-97370f5b04f7"). InnerVolumeSpecName "kube-api-access-56qm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:00.168296 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.168264 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/557af6d4-ae3b-4b48-ae87-75095fe70ec2-kube-api-access-q7j8l\" (UniqueName: \"kubernetes.io/projected/557af6d4-ae3b-4b48-ae87-75095fe70ec2-kube-api-access-q7j8l\") pod \"557af6d4-ae3b-4b48-ae87-75095fe70ec2\" (UID: \"557af6d4-ae3b-4b48-ae87-75095fe70ec2\") " Nov 03 02:17:00.168472 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.168348 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/557af6d4-ae3b-4b48-ae87-75095fe70ec2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/557af6d4-ae3b-4b48-ae87-75095fe70ec2-plugins-config-volume\") pod \"557af6d4-ae3b-4b48-ae87-75095fe70ec2\" (UID: \"557af6d4-ae3b-4b48-ae87-75095fe70ec2\") " Nov 03 02:17:00.168740 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.168719 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/557af6d4-ae3b-4b48-ae87-75095fe70ec2-plugins-config-volume" pod "557af6d4-ae3b-4b48-ae87-75095fe70ec2" (UID: "557af6d4-ae3b-4b48-ae87-75095fe70ec2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:17:00.168826 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.168813 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/557af6d4-ae3b-4b48-ae87-75095fe70ec2-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:00.168868 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.168827 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56qm6\" (UniqueName: \"kubernetes.io/projected/5472d731-9bca-404c-a5a4-97370f5b04f7-kube-api-access-56qm6\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:00.170419 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.170401 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/557af6d4-ae3b-4b48-ae87-75095fe70ec2-kube-api-access-q7j8l" pod "557af6d4-ae3b-4b48-ae87-75095fe70ec2" (UID: "557af6d4-ae3b-4b48-ae87-75095fe70ec2"). InnerVolumeSpecName "kube-api-access-q7j8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:00.270500 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.270434 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7j8l\" (UniqueName: \"kubernetes.io/projected/557af6d4-ae3b-4b48-ae87-75095fe70ec2-kube-api-access-q7j8l\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:00.432513 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.432471 5463 generic.go:356] "Generic (PLEG): container finished" podID="557af6d4-ae3b-4b48-ae87-75095fe70ec2" containerID="0e58fda70f3435c6a411bdfc3efc41add67e150892052ba85a5c1de17af938cb" exitCode=0 Nov 03 02:17:00.432686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.432532 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f" Nov 03 02:17:00.432686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.432567 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f" event={"ID":"557af6d4-ae3b-4b48-ae87-75095fe70ec2","Type":"ContainerDied","Data":"0e58fda70f3435c6a411bdfc3efc41add67e150892052ba85a5c1de17af938cb"} Nov 03 02:17:00.432686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.432629 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f" event={"ID":"557af6d4-ae3b-4b48-ae87-75095fe70ec2","Type":"ContainerDied","Data":"202598120f88a67c32aab02bd5a53e092447496705b79fac27f74f3f399e28df"} Nov 03 02:17:00.432686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.432655 5463 scope.go:122] "RemoveContainer" containerID="0e58fda70f3435c6a411bdfc3efc41add67e150892052ba85a5c1de17af938cb" Nov 03 02:17:00.434623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.434598 5463 generic.go:356] "Generic (PLEG): container finished" podID="5472d731-9bca-404c-a5a4-97370f5b04f7" containerID="6a3566f3b9a32c7cbc95e4991cb48a2dcdd961eb4805358de0414caa001f9274" exitCode=0 Nov 03 02:17:00.434806 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.434640 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd" Nov 03 02:17:00.434806 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.434736 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd" event={"ID":"5472d731-9bca-404c-a5a4-97370f5b04f7","Type":"ContainerDied","Data":"6a3566f3b9a32c7cbc95e4991cb48a2dcdd961eb4805358de0414caa001f9274"} Nov 03 02:17:00.434806 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.434772 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd" event={"ID":"5472d731-9bca-404c-a5a4-97370f5b04f7","Type":"ContainerDied","Data":"5509b534264cbd6a017257751f91657eee6f0044ed5becd0f677d0ff467008ca"} Nov 03 02:17:00.450692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.450611 5463 scope.go:122] "RemoveContainer" containerID="0e58fda70f3435c6a411bdfc3efc41add67e150892052ba85a5c1de17af938cb" Nov 03 02:17:00.453377 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:17:00.451184 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e58fda70f3435c6a411bdfc3efc41add67e150892052ba85a5c1de17af938cb\": container with ID starting with 0e58fda70f3435c6a411bdfc3efc41add67e150892052ba85a5c1de17af938cb not found: ID does not exist" containerID="0e58fda70f3435c6a411bdfc3efc41add67e150892052ba85a5c1de17af938cb" Nov 03 02:17:00.453594 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.453556 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e58fda70f3435c6a411bdfc3efc41add67e150892052ba85a5c1de17af938cb"} err="failed to get container status \"0e58fda70f3435c6a411bdfc3efc41add67e150892052ba85a5c1de17af938cb\": rpc error: code = NotFound desc = could not find container \"0e58fda70f3435c6a411bdfc3efc41add67e150892052ba85a5c1de17af938cb\": container with ID starting with 0e58fda70f3435c6a411bdfc3efc41add67e150892052ba85a5c1de17af938cb not found: ID does not exist" Nov 03 02:17:00.453594 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.453589 5463 scope.go:122] "RemoveContainer" containerID="6a3566f3b9a32c7cbc95e4991cb48a2dcdd961eb4805358de0414caa001f9274" Nov 03 02:17:00.465144 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.465113 5463 scope.go:122] "RemoveContainer" containerID="6a3566f3b9a32c7cbc95e4991cb48a2dcdd961eb4805358de0414caa001f9274" Nov 03 02:17:00.465479 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:17:00.465458 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a3566f3b9a32c7cbc95e4991cb48a2dcdd961eb4805358de0414caa001f9274\": container with ID starting with 6a3566f3b9a32c7cbc95e4991cb48a2dcdd961eb4805358de0414caa001f9274 not found: ID does not exist" containerID="6a3566f3b9a32c7cbc95e4991cb48a2dcdd961eb4805358de0414caa001f9274" Nov 03 02:17:00.465548 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.465486 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a3566f3b9a32c7cbc95e4991cb48a2dcdd961eb4805358de0414caa001f9274"} err="failed to get container status \"6a3566f3b9a32c7cbc95e4991cb48a2dcdd961eb4805358de0414caa001f9274\": rpc error: code = NotFound desc = could not find container \"6a3566f3b9a32c7cbc95e4991cb48a2dcdd961eb4805358de0414caa001f9274\": container with ID starting with 6a3566f3b9a32c7cbc95e4991cb48a2dcdd961eb4805358de0414caa001f9274 not found: ID does not exist" Nov 03 02:17:00.599149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.599062 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd"] Nov 03 02:17:00.648915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.648883 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c617de-6946d99c97-8b4sd"] Nov 03 02:17:00.758772 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.758739 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f"] Nov 03 02:17:00.762752 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:00.762719 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c617de-855bb784f5-v829f"] Nov 03 02:17:01.559831 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:01.559800 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:02.446678 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:02.446647 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5472d731-9bca-404c-a5a4-97370f5b04f7" path="/var/lib/kubelet/pods/5472d731-9bca-404c-a5a4-97370f5b04f7/volumes" Nov 03 02:17:02.447271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:02.447251 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="557af6d4-ae3b-4b48-ae87-75095fe70ec2" path="/var/lib/kubelet/pods/557af6d4-ae3b-4b48-ae87-75095fe70ec2/volumes" Nov 03 02:17:02.572997 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:02.572969 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:17:02.580242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:02.580219 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:17:02.788264 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:02.788153 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:03.700865 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:03.700829 5463 scope.go:122] "RemoveContainer" containerID="3fb5a1323a2e66cb2a6a9c0e77584c18c7cdd3f4c73f0468344d656d7fbea5ad" Nov 03 02:17:03.712726 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:03.712706 5463 scope.go:122] "RemoveContainer" containerID="0cfba01290b039c4902ed6b049e39f9411f18fc8eb7fa24bb60395651e1e8792" Nov 03 02:17:03.722519 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:03.722494 5463 scope.go:122] "RemoveContainer" containerID="72a3b169a664bba36d3d245a2102dd8aedcd3942b607980891940f0851c2e916" Nov 03 02:17:03.731787 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:03.731768 5463 scope.go:122] "RemoveContainer" containerID="1a3e4e42fc4071b31c87e101be01889db308976698bf97b6e58a427ebf2f01b7" Nov 03 02:17:03.741549 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:03.741525 5463 scope.go:122] "RemoveContainer" containerID="5c780aca00658e0539d6ada1850baddac90f69459fcbab862f2e1ab311ec8f9a" Nov 03 02:17:03.752417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:03.752395 5463 scope.go:122] "RemoveContainer" containerID="25803946466bfa1e80d722d603b66215e117706891df48ceb7b357f6bdfd6bfc" Nov 03 02:17:03.761141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:03.761122 5463 scope.go:122] "RemoveContainer" containerID="41a7835881912fc98ddffeed3855c4f8817c87fdc66a03266ab737c9a3d13943" Nov 03 02:17:03.769752 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:03.769731 5463 scope.go:122] "RemoveContainer" containerID="f813b120d921b04523a042d2e626ffee31630e7dff7b8b807606afd65d7ecf80" Nov 03 02:17:04.037100 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:04.036992 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:04.639899 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:04.639863 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:07.047132 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:07.047096 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78"] Nov 03 02:17:07.097364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:07.097331 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fcafcbb-svc78"] Nov 03 02:17:08.443790 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:08.443751 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbcf599a-af8c-494c-82d4-79ee96ea65c1" path="/var/lib/kubelet/pods/bbcf599a-af8c-494c-82d4-79ee96ea65c1/volumes" Nov 03 02:17:08.922265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:08.922213 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:10.160080 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:10.160049 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:10.444184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:10.444113 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb"] Nov 03 02:17:10.722307 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:10.722226 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:10.879220 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:10.878775 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:11.038453 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:11.038369 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:11.187744 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:11.187708 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:11.701446 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:11.701406 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:13.007759 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:17:13.007719 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:17:14.696972 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.696937 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw"] Nov 03 02:17:14.697572 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697559 5463 cpu_manager.go:414] "containerMap: removing container" podUID="32ef5394-19c0-4b33-84f6-2087dede03a0" containerName="istio-proxy" Nov 03 02:17:14.697631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697573 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ef5394-19c0-4b33-84f6-2087dede03a0" containerName="istio-proxy" Nov 03 02:17:14.697631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697587 5463 cpu_manager.go:414] "containerMap: removing container" podUID="13e969fe-f189-40f2-b929-e462f6d4b86b" containerName="istio-proxy" Nov 03 02:17:14.697631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697592 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13e969fe-f189-40f2-b929-e462f6d4b86b" containerName="istio-proxy" Nov 03 02:17:14.697631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697600 5463 cpu_manager.go:414] "containerMap: removing container" podUID="5472d731-9bca-404c-a5a4-97370f5b04f7" containerName="payload-processing" Nov 03 02:17:14.697631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697605 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5472d731-9bca-404c-a5a4-97370f5b04f7" containerName="payload-processing" Nov 03 02:17:14.697631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697618 5463 cpu_manager.go:414] "containerMap: removing container" podUID="cc2f2fb4-5a6f-4778-90e5-9a857204f663" containerName="istio-proxy" Nov 03 02:17:14.697631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697623 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc2f2fb4-5a6f-4778-90e5-9a857204f663" containerName="istio-proxy" Nov 03 02:17:14.697631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697630 5463 cpu_manager.go:414] "containerMap: removing container" podUID="9ec04276-3f4d-4bdf-96bf-37f93fd0913e" containerName="istio-proxy" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697635 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec04276-3f4d-4bdf-96bf-37f93fd0913e" containerName="istio-proxy" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697647 5463 cpu_manager.go:414] "containerMap: removing container" podUID="dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" containerName="istio-proxy" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697653 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" containerName="istio-proxy" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697662 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2746f922-b469-4383-90bb-3be08f9256d7" containerName="istio-proxy" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697668 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2746f922-b469-4383-90bb-3be08f9256d7" containerName="istio-proxy" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697674 5463 cpu_manager.go:414] "containerMap: removing container" podUID="731a025d-7302-4a22-8b43-70f96d8f05bb" containerName="istio-proxy" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697679 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731a025d-7302-4a22-8b43-70f96d8f05bb" containerName="istio-proxy" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697692 5463 cpu_manager.go:414] "containerMap: removing container" podUID="88714959-afe9-4294-b124-fca1f9d0c4f3" containerName="istio-proxy" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697698 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88714959-afe9-4294-b124-fca1f9d0c4f3" containerName="istio-proxy" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697704 5463 cpu_manager.go:414] "containerMap: removing container" podUID="8bd6ab07-9430-466d-9162-8e89ace4f015" containerName="payload-processing" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697709 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bd6ab07-9430-466d-9162-8e89ace4f015" containerName="payload-processing" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697717 5463 cpu_manager.go:414] "containerMap: removing container" podUID="9ec04276-3f4d-4bdf-96bf-37f93fd0913e" containerName="istio-proxy" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697721 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec04276-3f4d-4bdf-96bf-37f93fd0913e" containerName="istio-proxy" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697734 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2617013b-bed0-446d-a597-31bb323d1c6a" containerName="istio-proxy" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697738 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2617013b-bed0-446d-a597-31bb323d1c6a" containerName="istio-proxy" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697746 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2746f922-b469-4383-90bb-3be08f9256d7" containerName="istio-proxy" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697751 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2746f922-b469-4383-90bb-3be08f9256d7" containerName="istio-proxy" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697759 5463 cpu_manager.go:414] "containerMap: removing container" podUID="490ca339-509d-4f76-b2b7-6e6f57be24c7" containerName="payload-pre-processing" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697763 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="490ca339-509d-4f76-b2b7-6e6f57be24c7" containerName="payload-pre-processing" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697770 5463 cpu_manager.go:414] "containerMap: removing container" podUID="557af6d4-ae3b-4b48-ae87-75095fe70ec2" containerName="payload-pre-processing" Nov 03 02:17:14.697853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697775 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="557af6d4-ae3b-4b48-ae87-75095fe70ec2" containerName="payload-pre-processing" Nov 03 02:17:14.698486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697864 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc2f2fb4-5a6f-4778-90e5-9a857204f663" containerName="istio-proxy" Nov 03 02:17:14.698486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697872 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec04276-3f4d-4bdf-96bf-37f93fd0913e" containerName="istio-proxy" Nov 03 02:17:14.698486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697880 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfeb7b50-ad6f-4fa2-b672-cb47242deb1e" containerName="istio-proxy" Nov 03 02:17:14.698486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697886 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2746f922-b469-4383-90bb-3be08f9256d7" containerName="istio-proxy" Nov 03 02:17:14.698486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697892 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="731a025d-7302-4a22-8b43-70f96d8f05bb" containerName="istio-proxy" Nov 03 02:17:14.698486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697899 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="88714959-afe9-4294-b124-fca1f9d0c4f3" containerName="istio-proxy" Nov 03 02:17:14.698486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697905 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bd6ab07-9430-466d-9162-8e89ace4f015" containerName="payload-processing" Nov 03 02:17:14.698486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697913 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2617013b-bed0-446d-a597-31bb323d1c6a" containerName="istio-proxy" Nov 03 02:17:14.698486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697919 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="490ca339-509d-4f76-b2b7-6e6f57be24c7" containerName="payload-pre-processing" Nov 03 02:17:14.698486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697923 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="557af6d4-ae3b-4b48-ae87-75095fe70ec2" containerName="payload-pre-processing" Nov 03 02:17:14.698486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697932 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ef5394-19c0-4b33-84f6-2087dede03a0" containerName="istio-proxy" Nov 03 02:17:14.698486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697939 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="13e969fe-f189-40f2-b929-e462f6d4b86b" containerName="istio-proxy" Nov 03 02:17:14.698486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.697943 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="5472d731-9bca-404c-a5a4-97370f5b04f7" containerName="payload-processing" Nov 03 02:17:14.702082 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.702061 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw" Nov 03 02:17:14.753730 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.753707 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw"] Nov 03 02:17:14.755546 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.755524 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:17:14.755682 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.755537 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:17:14.800345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.800313 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:17:14.800494 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.800481 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:17:14.801648 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.801617 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2d1e8a-89d7-4729-8aed-0176001f4233-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92a46951-cbszw\" (UID: \"7a2d1e8a-89d7-4729-8aed-0176001f4233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw" Nov 03 02:17:14.801648 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.801692 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2d1e8a-89d7-4729-8aed-0176001f4233-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92a46951-cbszw\" (UID: \"7a2d1e8a-89d7-4729-8aed-0176001f4233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw" Nov 03 02:17:14.802519 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.801758 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s46lr\" (UniqueName: \"kubernetes.io/projected/7a2d1e8a-89d7-4729-8aed-0176001f4233-kube-api-access-s46lr\") pod \"maas-api-revoke-sub-92a46951-cbszw\" (UID: \"7a2d1e8a-89d7-4729-8aed-0176001f4233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw" Nov 03 02:17:14.902937 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.902906 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2d1e8a-89d7-4729-8aed-0176001f4233-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92a46951-cbszw\" (UID: \"7a2d1e8a-89d7-4729-8aed-0176001f4233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw" Nov 03 02:17:14.903112 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.902961 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2d1e8a-89d7-4729-8aed-0176001f4233-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92a46951-cbszw\" (UID: \"7a2d1e8a-89d7-4729-8aed-0176001f4233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw" Nov 03 02:17:14.903112 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.903022 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s46lr\" (UniqueName: \"kubernetes.io/projected/7a2d1e8a-89d7-4729-8aed-0176001f4233-kube-api-access-s46lr\") pod \"maas-api-revoke-sub-92a46951-cbszw\" (UID: \"7a2d1e8a-89d7-4729-8aed-0176001f4233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw" Nov 03 02:17:14.903851 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.903816 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2d1e8a-89d7-4729-8aed-0176001f4233-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92a46951-cbszw\" (UID: \"7a2d1e8a-89d7-4729-8aed-0176001f4233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw" Nov 03 02:17:14.905663 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.905638 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2d1e8a-89d7-4729-8aed-0176001f4233-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92a46951-cbszw\" (UID: \"7a2d1e8a-89d7-4729-8aed-0176001f4233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw" Nov 03 02:17:14.959840 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:14.959778 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s46lr\" (UniqueName: \"kubernetes.io/projected/7a2d1e8a-89d7-4729-8aed-0176001f4233-kube-api-access-s46lr\") pod \"maas-api-revoke-sub-92a46951-cbszw\" (UID: \"7a2d1e8a-89d7-4729-8aed-0176001f4233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw" Nov 03 02:17:15.014086 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:15.014061 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw" Nov 03 02:17:15.397083 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:17:15.397048 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2d1e8a_89d7_4729_8aed_0176001f4233.slice/crio-3bf992cf9fbe8e974c74d4ee997423c2b1003debed374c11e961b5bf093aa56a WatchSource:0}: Error finding container 3bf992cf9fbe8e974c74d4ee997423c2b1003debed374c11e961b5bf093aa56a: Status 404 returned error can't find the container with id 3bf992cf9fbe8e974c74d4ee997423c2b1003debed374c11e961b5bf093aa56a Nov 03 02:17:15.397347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:15.397328 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw"] Nov 03 02:17:15.400851 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:15.400820 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:15.444515 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:15.444456 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb" podUID="f4dc3ccd-f2c5-40f1-b834-f2a064032311" containerName="payload-processing" containerID="cri-o://e4e9d7e0bfc6f19f89fd4fc92485ffbb3bd436368220c27092a6524b0d812aff" gracePeriod=25 Nov 03 02:17:15.546433 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:15.546393 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw" event={"ID":"7a2d1e8a-89d7-4729-8aed-0176001f4233","Type":"ContainerStarted","Data":"3bf992cf9fbe8e974c74d4ee997423c2b1003debed374c11e961b5bf093aa56a"} Nov 03 02:17:15.717984 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:15.717954 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb" Nov 03 02:17:15.767236 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:15.766678 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:15.816936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:15.816898 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4dc3ccd-f2c5-40f1-b834-f2a064032311-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc3ccd-f2c5-40f1-b834-f2a064032311-plugins-config-volume\") pod \"f4dc3ccd-f2c5-40f1-b834-f2a064032311\" (UID: \"f4dc3ccd-f2c5-40f1-b834-f2a064032311\") " Nov 03 02:17:15.817327 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:15.817301 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4dc3ccd-f2c5-40f1-b834-f2a064032311-kube-api-access-x5xqv\" (UniqueName: \"kubernetes.io/projected/f4dc3ccd-f2c5-40f1-b834-f2a064032311-kube-api-access-x5xqv\") pod \"f4dc3ccd-f2c5-40f1-b834-f2a064032311\" (UID: \"f4dc3ccd-f2c5-40f1-b834-f2a064032311\") " Nov 03 02:17:15.819468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:15.819312 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4dc3ccd-f2c5-40f1-b834-f2a064032311-plugins-config-volume" pod "f4dc3ccd-f2c5-40f1-b834-f2a064032311" (UID: "f4dc3ccd-f2c5-40f1-b834-f2a064032311"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:17:15.829205 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:15.823605 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dc3ccd-f2c5-40f1-b834-f2a064032311-kube-api-access-x5xqv" pod "f4dc3ccd-f2c5-40f1-b834-f2a064032311" (UID: "f4dc3ccd-f2c5-40f1-b834-f2a064032311"). InnerVolumeSpecName "kube-api-access-x5xqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:15.924716 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:15.920231 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc3ccd-f2c5-40f1-b834-f2a064032311-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:15.924716 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:15.920268 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5xqv\" (UniqueName: \"kubernetes.io/projected/f4dc3ccd-f2c5-40f1-b834-f2a064032311-kube-api-access-x5xqv\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:16.226083 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:16.225699 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:16.554792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:16.554763 5463 generic.go:356] "Generic (PLEG): container finished" podID="f4dc3ccd-f2c5-40f1-b834-f2a064032311" containerID="e4e9d7e0bfc6f19f89fd4fc92485ffbb3bd436368220c27092a6524b0d812aff" exitCode=0 Nov 03 02:17:16.554940 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:16.554842 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb" Nov 03 02:17:16.554940 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:16.554855 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb" event={"ID":"f4dc3ccd-f2c5-40f1-b834-f2a064032311","Type":"ContainerDied","Data":"e4e9d7e0bfc6f19f89fd4fc92485ffbb3bd436368220c27092a6524b0d812aff"} Nov 03 02:17:16.554940 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:16.554925 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb" event={"ID":"f4dc3ccd-f2c5-40f1-b834-f2a064032311","Type":"ContainerDied","Data":"c40f43ff668820d334b38a92cb5bc96521963d8439ea43f594216d7cde8cda2c"} Nov 03 02:17:16.555134 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:16.554954 5463 scope.go:122] "RemoveContainer" containerID="e4e9d7e0bfc6f19f89fd4fc92485ffbb3bd436368220c27092a6524b0d812aff" Nov 03 02:17:16.556826 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:16.556803 5463 generic.go:356] "Generic (PLEG): container finished" podID="7a2d1e8a-89d7-4729-8aed-0176001f4233" containerID="b33e9dbef23ec658584cffdb36dc809c9b9bd6d60d597b994763896cbe974308" exitCode=0 Nov 03 02:17:16.556957 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:16.556916 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw" event={"ID":"7a2d1e8a-89d7-4729-8aed-0176001f4233","Type":"ContainerDied","Data":"b33e9dbef23ec658584cffdb36dc809c9b9bd6d60d597b994763896cbe974308"} Nov 03 02:17:16.566685 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:16.566647 5463 scope.go:122] "RemoveContainer" containerID="e4e9d7e0bfc6f19f89fd4fc92485ffbb3bd436368220c27092a6524b0d812aff" Nov 03 02:17:16.567038 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:17:16.567010 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e9d7e0bfc6f19f89fd4fc92485ffbb3bd436368220c27092a6524b0d812aff\": container with ID starting with e4e9d7e0bfc6f19f89fd4fc92485ffbb3bd436368220c27092a6524b0d812aff not found: ID does not exist" containerID="e4e9d7e0bfc6f19f89fd4fc92485ffbb3bd436368220c27092a6524b0d812aff" Nov 03 02:17:16.567172 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:16.567047 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e9d7e0bfc6f19f89fd4fc92485ffbb3bd436368220c27092a6524b0d812aff"} err="failed to get container status \"e4e9d7e0bfc6f19f89fd4fc92485ffbb3bd436368220c27092a6524b0d812aff\": rpc error: code = NotFound desc = could not find container \"e4e9d7e0bfc6f19f89fd4fc92485ffbb3bd436368220c27092a6524b0d812aff\": container with ID starting with e4e9d7e0bfc6f19f89fd4fc92485ffbb3bd436368220c27092a6524b0d812aff not found: ID does not exist" Nov 03 02:17:16.721029 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:16.720990 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb"] Nov 03 02:17:16.771425 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:16.771395 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a772c3-64f8cdd58f-cbhxb"] Nov 03 02:17:17.711951 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:17.711925 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw" Nov 03 02:17:17.848179 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:17.848088 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a2d1e8a-89d7-4729-8aed-0176001f4233-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2d1e8a-89d7-4729-8aed-0176001f4233-maas-api-xks-ca\") pod \"7a2d1e8a-89d7-4729-8aed-0176001f4233\" (UID: \"7a2d1e8a-89d7-4729-8aed-0176001f4233\") " Nov 03 02:17:17.848648 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:17.848221 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a2d1e8a-89d7-4729-8aed-0176001f4233-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2d1e8a-89d7-4729-8aed-0176001f4233-maas-api-openshift-ca\") pod \"7a2d1e8a-89d7-4729-8aed-0176001f4233\" (UID: \"7a2d1e8a-89d7-4729-8aed-0176001f4233\") " Nov 03 02:17:17.848648 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:17.848273 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2d1e8a-89d7-4729-8aed-0176001f4233-kube-api-access-s46lr\" (UniqueName: \"kubernetes.io/projected/7a2d1e8a-89d7-4729-8aed-0176001f4233-kube-api-access-s46lr\") pod \"7a2d1e8a-89d7-4729-8aed-0176001f4233\" (UID: \"7a2d1e8a-89d7-4729-8aed-0176001f4233\") " Nov 03 02:17:17.848828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:17.848657 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2d1e8a-89d7-4729-8aed-0176001f4233-maas-api-openshift-ca" pod "7a2d1e8a-89d7-4729-8aed-0176001f4233" (UID: "7a2d1e8a-89d7-4729-8aed-0176001f4233"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:17:17.849030 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:17.848992 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2d1e8a-89d7-4729-8aed-0176001f4233-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:17.850632 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:17.850610 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2d1e8a-89d7-4729-8aed-0176001f4233-kube-api-access-s46lr" pod "7a2d1e8a-89d7-4729-8aed-0176001f4233" (UID: "7a2d1e8a-89d7-4729-8aed-0176001f4233"). InnerVolumeSpecName "kube-api-access-s46lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:17.850738 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:17.850710 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2d1e8a-89d7-4729-8aed-0176001f4233-maas-api-xks-ca" pod "7a2d1e8a-89d7-4729-8aed-0176001f4233" (UID: "7a2d1e8a-89d7-4729-8aed-0176001f4233"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:17:17.950482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:17.950437 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2d1e8a-89d7-4729-8aed-0176001f4233-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:17.950482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:17.950472 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s46lr\" (UniqueName: \"kubernetes.io/projected/7a2d1e8a-89d7-4729-8aed-0176001f4233-kube-api-access-s46lr\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:18.443602 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:18.443561 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4dc3ccd-f2c5-40f1-b834-f2a064032311" path="/var/lib/kubelet/pods/f4dc3ccd-f2c5-40f1-b834-f2a064032311/volumes" Nov 03 02:17:18.572778 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:18.572737 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw" event={"ID":"7a2d1e8a-89d7-4729-8aed-0176001f4233","Type":"ContainerDied","Data":"3bf992cf9fbe8e974c74d4ee997423c2b1003debed374c11e961b5bf093aa56a"} Nov 03 02:17:18.572981 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:18.572828 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bf992cf9fbe8e974c74d4ee997423c2b1003debed374c11e961b5bf093aa56a" Nov 03 02:17:18.572981 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:18.572754 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw" Nov 03 02:17:19.116533 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:19.116482 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:19.347915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:19.347880 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:20.242057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:20.242025 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6"] Nov 03 02:17:20.242528 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:20.242299 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" podUID="8d2c4397-7a8c-41a5-9140-2079f97bd460" containerName="istio-proxy" containerID="cri-o://cbbbb46a6aa43d10924278e813715e80a88daa2a5e98c1df1f9a635c078c6c40" gracePeriod=30 Nov 03 02:17:20.292289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:20.292258 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58"] Nov 03 02:17:20.292562 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:20.292538 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" podUID="1e5aa4d8-7e07-4efe-96ef-93b35748423e" containerName="istio-proxy" containerID="cri-o://c96fe2e59e8dc0de2487dfcd40f84d03fbe88a4677750e2de31b404e8d4c99f0" gracePeriod=30 Nov 03 02:17:20.295355 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:20.295323 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5"] Nov 03 02:17:20.295525 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:20.295505 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" podUID="d52f99a6-11e1-4a01-b565-4d3730765a20" containerName="istio-proxy" containerID="cri-o://5eb2c2aec494e8610c7272d0be177ed492906458ed5049cae3a951f471f177b5" gracePeriod=30 Nov 03 02:17:20.342706 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:20.342680 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24"] Nov 03 02:17:20.342900 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:20.342877 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" podUID="831449be-effc-4efe-8005-f53a501168a1" containerName="istio-proxy" containerID="cri-o://e10202ed14343c408c1075548344dae7c28c27aaafcc7273f275135109c4cbc2" gracePeriod=30 Nov 03 02:17:20.877037 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:20.876999 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:24.168674 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:24.168642 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:24.863480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:24.863440 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:25.050323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.050289 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:25.302704 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.302667 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:25.485048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.482040 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf"] Nov 03 02:17:25.485048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.483518 5463 cpu_manager.go:414] "containerMap: removing container" podUID="7a2d1e8a-89d7-4729-8aed-0176001f4233" containerName="revoke-keys" Nov 03 02:17:25.485048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.483543 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2d1e8a-89d7-4729-8aed-0176001f4233" containerName="revoke-keys" Nov 03 02:17:25.485048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.483560 5463 cpu_manager.go:414] "containerMap: removing container" podUID="f4dc3ccd-f2c5-40f1-b834-f2a064032311" containerName="payload-processing" Nov 03 02:17:25.485048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.483569 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4dc3ccd-f2c5-40f1-b834-f2a064032311" containerName="payload-processing" Nov 03 02:17:25.485048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.483757 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec04276-3f4d-4bdf-96bf-37f93fd0913e" containerName="istio-proxy" Nov 03 02:17:25.485048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.483777 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2746f922-b469-4383-90bb-3be08f9256d7" containerName="istio-proxy" Nov 03 02:17:25.485048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.483795 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2d1e8a-89d7-4729-8aed-0176001f4233" containerName="revoke-keys" Nov 03 02:17:25.485048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.483803 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4dc3ccd-f2c5-40f1-b834-f2a064032311" containerName="payload-processing" Nov 03 02:17:25.493410 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.493385 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf" Nov 03 02:17:25.533179 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.533097 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf"] Nov 03 02:17:25.544225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.544100 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9"] Nov 03 02:17:25.552077 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.552051 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.581134 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.580401 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:17:25.585245 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.583099 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-701ba30f\"" Nov 03 02:17:25.585245 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.583322 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-701ba30f-dockercfg-p5xnw\"" Nov 03 02:17:25.585245 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.584685 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a22f93-19ab-4222-9f41-9ad94cc2642c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf\" (UID: \"c6a22f93-19ab-4222-9f41-9ad94cc2642c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf" Nov 03 02:17:25.585245 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.585074 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29bht\" (UniqueName: \"kubernetes.io/projected/c6a22f93-19ab-4222-9f41-9ad94cc2642c-kube-api-access-29bht\") pod \"payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf\" (UID: \"c6a22f93-19ab-4222-9f41-9ad94cc2642c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf" Nov 03 02:17:25.639900 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.639871 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9"] Nov 03 02:17:25.643430 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.643402 5463 generic.go:356] "Generic (PLEG): container finished" podID="d52f99a6-11e1-4a01-b565-4d3730765a20" containerID="5eb2c2aec494e8610c7272d0be177ed492906458ed5049cae3a951f471f177b5" exitCode=0 Nov 03 02:17:25.643661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.643490 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" event={"ID":"d52f99a6-11e1-4a01-b565-4d3730765a20","Type":"ContainerDied","Data":"5eb2c2aec494e8610c7272d0be177ed492906458ed5049cae3a951f471f177b5"} Nov 03 02:17:25.647672 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.647569 5463 generic.go:356] "Generic (PLEG): container finished" podID="1e5aa4d8-7e07-4efe-96ef-93b35748423e" containerID="c96fe2e59e8dc0de2487dfcd40f84d03fbe88a4677750e2de31b404e8d4c99f0" exitCode=0 Nov 03 02:17:25.647672 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.647605 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f9b509-openshift-default-dockercfg-jqfpx\"" Nov 03 02:17:25.647945 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.647823 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" event={"ID":"1e5aa4d8-7e07-4efe-96ef-93b35748423e","Type":"ContainerDied","Data":"c96fe2e59e8dc0de2487dfcd40f84d03fbe88a4677750e2de31b404e8d4c99f0"} Nov 03 02:17:25.658492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.653662 5463 generic.go:356] "Generic (PLEG): container finished" podID="831449be-effc-4efe-8005-f53a501168a1" containerID="e10202ed14343c408c1075548344dae7c28c27aaafcc7273f275135109c4cbc2" exitCode=0 Nov 03 02:17:25.658492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.653781 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" event={"ID":"831449be-effc-4efe-8005-f53a501168a1","Type":"ContainerDied","Data":"e10202ed14343c408c1075548344dae7c28c27aaafcc7273f275135109c4cbc2"} Nov 03 02:17:25.658492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.653854 5463 scope.go:122] "RemoveContainer" containerID="4533b5687e519f60e97c2829a3c3aa50ae280db490914a70217ea31ef604248b" Nov 03 02:17:25.658492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.657227 5463 generic.go:356] "Generic (PLEG): container finished" podID="8d2c4397-7a8c-41a5-9140-2079f97bd460" containerID="cbbbb46a6aa43d10924278e813715e80a88daa2a5e98c1df1f9a635c078c6c40" exitCode=0 Nov 03 02:17:25.658492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.657329 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" event={"ID":"8d2c4397-7a8c-41a5-9140-2079f97bd460","Type":"ContainerDied","Data":"cbbbb46a6aa43d10924278e813715e80a88daa2a5e98c1df1f9a635c078c6c40"} Nov 03 02:17:25.658492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.657389 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" event={"ID":"8d2c4397-7a8c-41a5-9140-2079f97bd460","Type":"ContainerDied","Data":"11506ee390343020b2cd9a2ce94c0a6efbdccfd6525e47fa4c710a8d43f5ea72"} Nov 03 02:17:25.658492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.657488 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6" Nov 03 02:17:25.677240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.672080 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:25.681251 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.681231 5463 scope.go:122] "RemoveContainer" containerID="cbbbb46a6aa43d10924278e813715e80a88daa2a5e98c1df1f9a635c078c6c40" Nov 03 02:17:25.687692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.686031 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-envoy\") pod \"8d2c4397-7a8c-41a5-9140-2079f97bd460\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " Nov 03 02:17:25.687692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.686117 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-workload-socket\") pod \"8d2c4397-7a8c-41a5-9140-2079f97bd460\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " Nov 03 02:17:25.687692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.686244 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-token\" (UniqueName: \"kubernetes.io/projected/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-token\") pod \"8d2c4397-7a8c-41a5-9140-2079f97bd460\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " Nov 03 02:17:25.687692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.686304 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-podinfo\") pod \"8d2c4397-7a8c-41a5-9140-2079f97bd460\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " Nov 03 02:17:25.687692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.686456 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d2c4397-7a8c-41a5-9140-2079f97bd460-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d2c4397-7a8c-41a5-9140-2079f97bd460-istiod-ca-cert\") pod \"8d2c4397-7a8c-41a5-9140-2079f97bd460\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " Nov 03 02:17:25.687692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.686505 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-workload-certs\") pod \"8d2c4397-7a8c-41a5-9140-2079f97bd460\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " Nov 03 02:17:25.687692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.686583 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2c4397-7a8c-41a5-9140-2079f97bd460-kube-api-access-7jx56\" (UniqueName: \"kubernetes.io/projected/8d2c4397-7a8c-41a5-9140-2079f97bd460-kube-api-access-7jx56\") pod \"8d2c4397-7a8c-41a5-9140-2079f97bd460\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " Nov 03 02:17:25.687692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.686618 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-credential-socket\") pod \"8d2c4397-7a8c-41a5-9140-2079f97bd460\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " Nov 03 02:17:25.687692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.686657 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-data\") pod \"8d2c4397-7a8c-41a5-9140-2079f97bd460\" (UID: \"8d2c4397-7a8c-41a5-9140-2079f97bd460\") " Nov 03 02:17:25.687692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.686856 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-envoy\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.687692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.686960 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a22f93-19ab-4222-9f41-9ad94cc2642c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf\" (UID: \"c6a22f93-19ab-4222-9f41-9ad94cc2642c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf" Nov 03 02:17:25.687692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.687014 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istiod-ca-cert\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.687692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.687165 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-workload-socket\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.687692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.687229 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-workload-certs\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.687692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.687261 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62z22\" (UniqueName: \"kubernetes.io/projected/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-kube-api-access-62z22\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.687692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.687320 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-token\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.687692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.687409 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-podinfo\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.687692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.687481 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-data\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.687692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.687551 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-credential-socket\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.687692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.687579 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29bht\" (UniqueName: \"kubernetes.io/projected/c6a22f93-19ab-4222-9f41-9ad94cc2642c-kube-api-access-29bht\") pod \"payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf\" (UID: \"c6a22f93-19ab-4222-9f41-9ad94cc2642c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf" Nov 03 02:17:25.696866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.688954 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-workload-socket" pod "8d2c4397-7a8c-41a5-9140-2079f97bd460" (UID: "8d2c4397-7a8c-41a5-9140-2079f97bd460"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:25.696866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.690549 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-credential-socket" pod "8d2c4397-7a8c-41a5-9140-2079f97bd460" (UID: "8d2c4397-7a8c-41a5-9140-2079f97bd460"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:25.696866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.690615 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-data" pod "8d2c4397-7a8c-41a5-9140-2079f97bd460" (UID: "8d2c4397-7a8c-41a5-9140-2079f97bd460"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:25.696866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.690762 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-workload-certs" pod "8d2c4397-7a8c-41a5-9140-2079f97bd460" (UID: "8d2c4397-7a8c-41a5-9140-2079f97bd460"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:25.696866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.691732 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a22f93-19ab-4222-9f41-9ad94cc2642c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf\" (UID: \"c6a22f93-19ab-4222-9f41-9ad94cc2642c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf" Nov 03 02:17:25.696866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.692951 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2c4397-7a8c-41a5-9140-2079f97bd460-istiod-ca-cert" pod "8d2c4397-7a8c-41a5-9140-2079f97bd460" (UID: "8d2c4397-7a8c-41a5-9140-2079f97bd460"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:17:25.696866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.694428 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-podinfo" pod "8d2c4397-7a8c-41a5-9140-2079f97bd460" (UID: "8d2c4397-7a8c-41a5-9140-2079f97bd460"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:17:25.717955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.717778 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:17:25.721230 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.718709 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-token" pod "8d2c4397-7a8c-41a5-9140-2079f97bd460" (UID: "8d2c4397-7a8c-41a5-9140-2079f97bd460"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:25.721230 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.720802 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2c4397-7a8c-41a5-9140-2079f97bd460-kube-api-access-7jx56" pod "8d2c4397-7a8c-41a5-9140-2079f97bd460" (UID: "8d2c4397-7a8c-41a5-9140-2079f97bd460"). InnerVolumeSpecName "kube-api-access-7jx56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:25.722366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.722320 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-envoy" pod "8d2c4397-7a8c-41a5-9140-2079f97bd460" (UID: "8d2c4397-7a8c-41a5-9140-2079f97bd460"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:25.724366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.724341 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p"] Nov 03 02:17:25.725382 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.725361 5463 cpu_manager.go:414] "containerMap: removing container" podUID="1e5aa4d8-7e07-4efe-96ef-93b35748423e" containerName="istio-proxy" Nov 03 02:17:25.725480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.725383 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5aa4d8-7e07-4efe-96ef-93b35748423e" containerName="istio-proxy" Nov 03 02:17:25.725480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.725424 5463 cpu_manager.go:414] "containerMap: removing container" podUID="8d2c4397-7a8c-41a5-9140-2079f97bd460" containerName="istio-proxy" Nov 03 02:17:25.725480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.725433 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2c4397-7a8c-41a5-9140-2079f97bd460" containerName="istio-proxy" Nov 03 02:17:25.725642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.725582 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2c4397-7a8c-41a5-9140-2079f97bd460" containerName="istio-proxy" Nov 03 02:17:25.725642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.725604 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5aa4d8-7e07-4efe-96ef-93b35748423e" containerName="istio-proxy" Nov 03 02:17:25.734141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.729525 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p" Nov 03 02:17:25.753574 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.753548 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29bht\" (UniqueName: \"kubernetes.io/projected/c6a22f93-19ab-4222-9f41-9ad94cc2642c-kube-api-access-29bht\") pod \"payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf\" (UID: \"c6a22f93-19ab-4222-9f41-9ad94cc2642c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf" Nov 03 02:17:25.753707 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.753686 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p"] Nov 03 02:17:25.775807 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.769348 5463 scope.go:122] "RemoveContainer" containerID="cbbbb46a6aa43d10924278e813715e80a88daa2a5e98c1df1f9a635c078c6c40" Nov 03 02:17:25.776482 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:17:25.776377 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbbbb46a6aa43d10924278e813715e80a88daa2a5e98c1df1f9a635c078c6c40\": container with ID starting with cbbbb46a6aa43d10924278e813715e80a88daa2a5e98c1df1f9a635c078c6c40 not found: ID does not exist" containerID="cbbbb46a6aa43d10924278e813715e80a88daa2a5e98c1df1f9a635c078c6c40" Nov 03 02:17:25.776482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.776416 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbbbb46a6aa43d10924278e813715e80a88daa2a5e98c1df1f9a635c078c6c40"} err="failed to get container status \"cbbbb46a6aa43d10924278e813715e80a88daa2a5e98c1df1f9a635c078c6c40\": rpc error: code = NotFound desc = could not find container \"cbbbb46a6aa43d10924278e813715e80a88daa2a5e98c1df1f9a635c078c6c40\": container with ID starting with cbbbb46a6aa43d10924278e813715e80a88daa2a5e98c1df1f9a635c078c6c40 not found: ID does not exist" Nov 03 02:17:25.790139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.789326 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-workload-socket\") pod \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " Nov 03 02:17:25.790139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.789417 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-envoy\") pod \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " Nov 03 02:17:25.790139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.789502 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-workload-certs\") pod \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " Nov 03 02:17:25.790139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.789570 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-credential-socket\") pod \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " Nov 03 02:17:25.790139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.789602 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-podinfo\") pod \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " Nov 03 02:17:25.790139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.789679 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-token\" (UniqueName: \"kubernetes.io/projected/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-token\") pod \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " Nov 03 02:17:25.790139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.789797 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5aa4d8-7e07-4efe-96ef-93b35748423e-kube-api-access-t4lkx\" (UniqueName: \"kubernetes.io/projected/1e5aa4d8-7e07-4efe-96ef-93b35748423e-kube-api-access-t4lkx\") pod \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " Nov 03 02:17:25.790139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.789884 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istiod-ca-cert\") pod \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " Nov 03 02:17:25.790139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.789935 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-data\") pod \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\" (UID: \"1e5aa4d8-7e07-4efe-96ef-93b35748423e\") " Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.790628 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msslv\" (UniqueName: \"kubernetes.io/projected/bd79820a-5bdb-449a-9e7f-559755e6e9ee-kube-api-access-msslv\") pod \"payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p\" (UID: \"bd79820a-5bdb-449a-9e7f-559755e6e9ee\") " pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.790700 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istiod-ca-cert\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.790834 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-workload-socket\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.790876 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-workload-certs\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.790906 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62z22\" (UniqueName: \"kubernetes.io/projected/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-kube-api-access-62z22\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.790958 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-token\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.791029 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-podinfo\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.791669 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istiod-ca-cert" pod "1e5aa4d8-7e07-4efe-96ef-93b35748423e" (UID: "1e5aa4d8-7e07-4efe-96ef-93b35748423e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.791090 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-data\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.792580 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-credential-socket\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.792625 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-envoy\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.792671 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd79820a-5bdb-449a-9e7f-559755e6e9ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p\" (UID: \"bd79820a-5bdb-449a-9e7f-559755e6e9ee\") " pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.792799 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d2c4397-7a8c-41a5-9140-2079f97bd460-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.792816 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.792831 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jx56\" (UniqueName: \"kubernetes.io/projected/8d2c4397-7a8c-41a5-9140-2079f97bd460-kube-api-access-7jx56\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.792845 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.792861 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.792873 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.792886 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2c4397-7a8c-41a5-9140-2079f97bd460-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.792900 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.792914 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.792929 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d2c4397-7a8c-41a5-9140-2079f97bd460-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.793827 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istiod-ca-cert\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.794137 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-workload-socket\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.794406 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-workload-certs\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.791936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.795969 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5aa4d8-7e07-4efe-96ef-93b35748423e-kube-api-access-t4lkx" pod "1e5aa4d8-7e07-4efe-96ef-93b35748423e" (UID: "1e5aa4d8-7e07-4efe-96ef-93b35748423e"). InnerVolumeSpecName "kube-api-access-t4lkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:25.810222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.796475 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-workload-socket" pod "1e5aa4d8-7e07-4efe-96ef-93b35748423e" (UID: "1e5aa4d8-7e07-4efe-96ef-93b35748423e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:25.810222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.797257 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-credential-socket\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.810222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.797529 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-data\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.810222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.797755 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-data" pod "1e5aa4d8-7e07-4efe-96ef-93b35748423e" (UID: "1e5aa4d8-7e07-4efe-96ef-93b35748423e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:25.810222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.798068 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-workload-certs" pod "1e5aa4d8-7e07-4efe-96ef-93b35748423e" (UID: "1e5aa4d8-7e07-4efe-96ef-93b35748423e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:25.810222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.798297 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-credential-socket" pod "1e5aa4d8-7e07-4efe-96ef-93b35748423e" (UID: "1e5aa4d8-7e07-4efe-96ef-93b35748423e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:25.810222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.808682 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-podinfo\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.810222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.808817 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-token" pod "1e5aa4d8-7e07-4efe-96ef-93b35748423e" (UID: "1e5aa4d8-7e07-4efe-96ef-93b35748423e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:25.810222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.809027 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-envoy" pod "1e5aa4d8-7e07-4efe-96ef-93b35748423e" (UID: "1e5aa4d8-7e07-4efe-96ef-93b35748423e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:25.810222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.809310 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-envoy\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.823476 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.810976 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-podinfo" pod "1e5aa4d8-7e07-4efe-96ef-93b35748423e" (UID: "1e5aa4d8-7e07-4efe-96ef-93b35748423e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:17:25.823476 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.816496 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:17:25.823476 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.822348 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:17:25.834221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.828898 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf" Nov 03 02:17:25.883359 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.870801 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-token\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.883359 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.871328 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:25.887401 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.887341 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62z22\" (UniqueName: \"kubernetes.io/projected/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-kube-api-access-62z22\") pod \"e2e-api-b-f9b509-openshift-default-9f6576965-j84n9\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.893901 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-workload-socket\") pod \"831449be-effc-4efe-8005-f53a501168a1\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.893958 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-token\" (UniqueName: \"kubernetes.io/projected/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-token\") pod \"d52f99a6-11e1-4a01-b565-4d3730765a20\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.894055 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-istio-envoy\") pod \"831449be-effc-4efe-8005-f53a501168a1\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.894099 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/831449be-effc-4efe-8005-f53a501168a1-kube-api-access-76sw2\" (UniqueName: \"kubernetes.io/projected/831449be-effc-4efe-8005-f53a501168a1-kube-api-access-76sw2\") pod \"831449be-effc-4efe-8005-f53a501168a1\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.894151 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/831449be-effc-4efe-8005-f53a501168a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/831449be-effc-4efe-8005-f53a501168a1-istio-podinfo\") pod \"831449be-effc-4efe-8005-f53a501168a1\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.894339 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/831449be-effc-4efe-8005-f53a501168a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/831449be-effc-4efe-8005-f53a501168a1-istiod-ca-cert\") pod \"831449be-effc-4efe-8005-f53a501168a1\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.894397 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-envoy\") pod \"d52f99a6-11e1-4a01-b565-4d3730765a20\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.894433 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-credential-socket\") pod \"d52f99a6-11e1-4a01-b565-4d3730765a20\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.894473 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-credential-socket\") pod \"831449be-effc-4efe-8005-f53a501168a1\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.894507 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d52f99a6-11e1-4a01-b565-4d3730765a20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52f99a6-11e1-4a01-b565-4d3730765a20-istiod-ca-cert\") pod \"d52f99a6-11e1-4a01-b565-4d3730765a20\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.894543 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-istio-data\") pod \"831449be-effc-4efe-8005-f53a501168a1\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.894600 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-data\") pod \"d52f99a6-11e1-4a01-b565-4d3730765a20\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.894632 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-workload-certs\") pod \"d52f99a6-11e1-4a01-b565-4d3730765a20\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.894713 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52f99a6-11e1-4a01-b565-4d3730765a20-kube-api-access-stnqc\" (UniqueName: \"kubernetes.io/projected/d52f99a6-11e1-4a01-b565-4d3730765a20-kube-api-access-stnqc\") pod \"d52f99a6-11e1-4a01-b565-4d3730765a20\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.894774 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/831449be-effc-4efe-8005-f53a501168a1-istio-token\" (UniqueName: \"kubernetes.io/projected/831449be-effc-4efe-8005-f53a501168a1-istio-token\") pod \"831449be-effc-4efe-8005-f53a501168a1\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.894811 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-workload-socket\") pod \"d52f99a6-11e1-4a01-b565-4d3730765a20\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.894866 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-podinfo\") pod \"d52f99a6-11e1-4a01-b565-4d3730765a20\" (UID: \"d52f99a6-11e1-4a01-b565-4d3730765a20\") " Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.894961 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-workload-certs\") pod \"831449be-effc-4efe-8005-f53a501168a1\" (UID: \"831449be-effc-4efe-8005-f53a501168a1\") " Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.895487 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd79820a-5bdb-449a-9e7f-559755e6e9ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p\" (UID: \"bd79820a-5bdb-449a-9e7f-559755e6e9ee\") " pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p" Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.895566 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msslv\" (UniqueName: \"kubernetes.io/projected/bd79820a-5bdb-449a-9e7f-559755e6e9ee-kube-api-access-msslv\") pod \"payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p\" (UID: \"bd79820a-5bdb-449a-9e7f-559755e6e9ee\") " pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p" Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.895735 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.895750 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.895764 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.895779 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.895793 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4lkx\" (UniqueName: \"kubernetes.io/projected/1e5aa4d8-7e07-4efe-96ef-93b35748423e-kube-api-access-t4lkx\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.895810 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.895826 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:25.896304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.895840 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5aa4d8-7e07-4efe-96ef-93b35748423e-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:25.898183 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.897872 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-workload-socket" pod "831449be-effc-4efe-8005-f53a501168a1" (UID: "831449be-effc-4efe-8005-f53a501168a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:25.900985 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.900631 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-token" pod "d52f99a6-11e1-4a01-b565-4d3730765a20" (UID: "d52f99a6-11e1-4a01-b565-4d3730765a20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:25.901239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.901174 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-istio-data" pod "831449be-effc-4efe-8005-f53a501168a1" (UID: "831449be-effc-4efe-8005-f53a501168a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:25.905665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.902426 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:25.905665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.905168 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-credential-socket" pod "831449be-effc-4efe-8005-f53a501168a1" (UID: "831449be-effc-4efe-8005-f53a501168a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:25.905665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.905356 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-credential-socket" pod "d52f99a6-11e1-4a01-b565-4d3730765a20" (UID: "d52f99a6-11e1-4a01-b565-4d3730765a20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:25.909399 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.906531 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/831449be-effc-4efe-8005-f53a501168a1-istiod-ca-cert" pod "831449be-effc-4efe-8005-f53a501168a1" (UID: "831449be-effc-4efe-8005-f53a501168a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:17:25.909399 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.908572 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52f99a6-11e1-4a01-b565-4d3730765a20-istiod-ca-cert" pod "d52f99a6-11e1-4a01-b565-4d3730765a20" (UID: "d52f99a6-11e1-4a01-b565-4d3730765a20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:17:25.909399 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.908846 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-workload-certs" pod "d52f99a6-11e1-4a01-b565-4d3730765a20" (UID: "d52f99a6-11e1-4a01-b565-4d3730765a20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:25.909399 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.909008 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-data" pod "d52f99a6-11e1-4a01-b565-4d3730765a20" (UID: "d52f99a6-11e1-4a01-b565-4d3730765a20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:25.909399 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.909154 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-workload-socket" pod "d52f99a6-11e1-4a01-b565-4d3730765a20" (UID: "d52f99a6-11e1-4a01-b565-4d3730765a20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:25.915228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.910502 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-istio-envoy" pod "831449be-effc-4efe-8005-f53a501168a1" (UID: "831449be-effc-4efe-8005-f53a501168a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:25.915228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.914101 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-workload-certs" pod "831449be-effc-4efe-8005-f53a501168a1" (UID: "831449be-effc-4efe-8005-f53a501168a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:25.915228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.915122 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd79820a-5bdb-449a-9e7f-559755e6e9ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p\" (UID: \"bd79820a-5bdb-449a-9e7f-559755e6e9ee\") " pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p" Nov 03 02:17:25.923743 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.923702 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/831449be-effc-4efe-8005-f53a501168a1-istio-podinfo" pod "831449be-effc-4efe-8005-f53a501168a1" (UID: "831449be-effc-4efe-8005-f53a501168a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:17:25.926809 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.926491 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/831449be-effc-4efe-8005-f53a501168a1-kube-api-access-76sw2" pod "831449be-effc-4efe-8005-f53a501168a1" (UID: "831449be-effc-4efe-8005-f53a501168a1"). InnerVolumeSpecName "kube-api-access-76sw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:25.926809 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.926648 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-envoy" pod "d52f99a6-11e1-4a01-b565-4d3730765a20" (UID: "d52f99a6-11e1-4a01-b565-4d3730765a20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:25.926809 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.926777 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/831449be-effc-4efe-8005-f53a501168a1-istio-token" pod "831449be-effc-4efe-8005-f53a501168a1" (UID: "831449be-effc-4efe-8005-f53a501168a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:25.928363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.928324 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52f99a6-11e1-4a01-b565-4d3730765a20-kube-api-access-stnqc" pod "d52f99a6-11e1-4a01-b565-4d3730765a20" (UID: "d52f99a6-11e1-4a01-b565-4d3730765a20"). InnerVolumeSpecName "kube-api-access-stnqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:25.934601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.934573 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-podinfo" pod "d52f99a6-11e1-4a01-b565-4d3730765a20" (UID: "d52f99a6-11e1-4a01-b565-4d3730765a20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:17:25.959971 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.959928 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msslv\" (UniqueName: \"kubernetes.io/projected/bd79820a-5bdb-449a-9e7f-559755e6e9ee-kube-api-access-msslv\") pod \"payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p\" (UID: \"bd79820a-5bdb-449a-9e7f-559755e6e9ee\") " pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p" Nov 03 02:17:26.007312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.998877 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stnqc\" (UniqueName: \"kubernetes.io/projected/d52f99a6-11e1-4a01-b565-4d3730765a20-kube-api-access-stnqc\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:26.007312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.998910 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/831449be-effc-4efe-8005-f53a501168a1-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:26.007312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.998920 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:26.007312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.998930 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:26.007312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.998939 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:26.007312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.998951 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:26.007312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.998961 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:26.007312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.998974 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:26.007312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.998990 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76sw2\" (UniqueName: \"kubernetes.io/projected/831449be-effc-4efe-8005-f53a501168a1-kube-api-access-76sw2\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:26.007312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.998999 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/831449be-effc-4efe-8005-f53a501168a1-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:26.007312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.999009 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/831449be-effc-4efe-8005-f53a501168a1-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:26.007312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.999036 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:26.007312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.999054 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:26.007312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.999064 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:26.007312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.999074 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52f99a6-11e1-4a01-b565-4d3730765a20-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:26.007312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.999083 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/831449be-effc-4efe-8005-f53a501168a1-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:26.007312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.999096 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:26.007312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:25.999105 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52f99a6-11e1-4a01-b565-4d3730765a20-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:26.066711 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.066398 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p" Nov 03 02:17:26.080663 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.080596 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:26.293807 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:17:26.290639 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a22f93_19ab_4222_9f41_9ad94cc2642c.slice/crio-300e51bf46a42546a66ad7d22db1085a30e8347249dcd4c63b2a9eab8d3cf241 WatchSource:0}: Error finding container 300e51bf46a42546a66ad7d22db1085a30e8347249dcd4c63b2a9eab8d3cf241: Status 404 returned error can't find the container with id 300e51bf46a42546a66ad7d22db1085a30e8347249dcd4c63b2a9eab8d3cf241 Nov 03 02:17:26.294904 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.294875 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf"] Nov 03 02:17:26.415873 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:17:26.415833 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84137e59_ff98_4062_b0d7_7fa4f2fbdc9e.slice/crio-01882f8ba45b378bcf58ac613d04d91e4f18d8640b19842c0ce9d24c7bbee93b WatchSource:0}: Error finding container 01882f8ba45b378bcf58ac613d04d91e4f18d8640b19842c0ce9d24c7bbee93b: Status 404 returned error can't find the container with id 01882f8ba45b378bcf58ac613d04d91e4f18d8640b19842c0ce9d24c7bbee93b Nov 03 02:17:26.418880 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.418840 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:17:26.418957 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.418928 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:17:26.418992 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.418973 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:17:26.419588 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.419571 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9"] Nov 03 02:17:26.496867 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:17:26.496835 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd79820a_5bdb_449a_9e7f_559755e6e9ee.slice/crio-56f8cfefb5ea67fe202b52732cfafc2c2afe10ee942beffd55135a21c9de35be WatchSource:0}: Error finding container 56f8cfefb5ea67fe202b52732cfafc2c2afe10ee942beffd55135a21c9de35be: Status 404 returned error can't find the container with id 56f8cfefb5ea67fe202b52732cfafc2c2afe10ee942beffd55135a21c9de35be Nov 03 02:17:26.499797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.499769 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p"] Nov 03 02:17:26.550375 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.550305 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6"] Nov 03 02:17:26.551518 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.551484 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-sxtg6"] Nov 03 02:17:26.670549 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.670513 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p" event={"ID":"bd79820a-5bdb-449a-9e7f-559755e6e9ee","Type":"ContainerStarted","Data":"90dcccbca5425e9820bad1eff04179c7c0baf09ee59dd793280f9198c94d8144"} Nov 03 02:17:26.670720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.670572 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p" event={"ID":"bd79820a-5bdb-449a-9e7f-559755e6e9ee","Type":"ContainerStarted","Data":"56f8cfefb5ea67fe202b52732cfafc2c2afe10ee942beffd55135a21c9de35be"} Nov 03 02:17:26.670720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.670599 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p" Nov 03 02:17:26.672166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.672143 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" event={"ID":"d52f99a6-11e1-4a01-b565-4d3730765a20","Type":"ContainerDied","Data":"71fd01aaf34fa90c2b0cc37cc702fcf08ccdf96f59edeb7c5790acd752b4bce1"} Nov 03 02:17:26.672384 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.672151 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5" Nov 03 02:17:26.672517 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.672218 5463 scope.go:122] "RemoveContainer" containerID="5eb2c2aec494e8610c7272d0be177ed492906458ed5049cae3a951f471f177b5" Nov 03 02:17:26.673851 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.673827 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" event={"ID":"1e5aa4d8-7e07-4efe-96ef-93b35748423e","Type":"ContainerDied","Data":"1841c2b3f15a90f77c9968bc2161869fad99127a7b67fd599ed07fdaea54e08c"} Nov 03 02:17:26.673959 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.673941 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58" Nov 03 02:17:26.677459 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.676827 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf" event={"ID":"c6a22f93-19ab-4222-9f41-9ad94cc2642c","Type":"ContainerStarted","Data":"452106c15460077f1f55472c31392b1fcf532e8e8420f59659fd736dff6e7ce0"} Nov 03 02:17:26.677459 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.676884 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf" Nov 03 02:17:26.677459 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.676909 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf" event={"ID":"c6a22f93-19ab-4222-9f41-9ad94cc2642c","Type":"ContainerStarted","Data":"300e51bf46a42546a66ad7d22db1085a30e8347249dcd4c63b2a9eab8d3cf241"} Nov 03 02:17:26.682233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.679404 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" event={"ID":"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e","Type":"ContainerStarted","Data":"e1667115830b7fe3fae41cf011b4b63a9805ba45cb461ee9652d11320dbda32c"} Nov 03 02:17:26.682233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.679448 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" event={"ID":"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e","Type":"ContainerStarted","Data":"01882f8ba45b378bcf58ac613d04d91e4f18d8640b19842c0ce9d24c7bbee93b"} Nov 03 02:17:26.682233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.680925 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" event={"ID":"831449be-effc-4efe-8005-f53a501168a1","Type":"ContainerDied","Data":"5a87672cca5cee2f2d0608232454a788a7ff680c4269a0a1a8f124878cc20851"} Nov 03 02:17:26.682233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.680982 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24" Nov 03 02:17:26.686247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.686228 5463 scope.go:122] "RemoveContainer" containerID="c96fe2e59e8dc0de2487dfcd40f84d03fbe88a4677750e2de31b404e8d4c99f0" Nov 03 02:17:26.695731 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.695712 5463 scope.go:122] "RemoveContainer" containerID="e10202ed14343c408c1075548344dae7c28c27aaafcc7273f275135109c4cbc2" Nov 03 02:17:26.785811 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.785751 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p" podStartSLOduration=1.785732004 podStartE2EDuration="1.785732004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:17:26.783626489 +0000 UTC m=+1528.467735645" watchObservedRunningTime="2026-11-03 02:17:26.785732004 +0000 UTC m=+1528.469841162" Nov 03 02:17:26.898315 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.898257 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf" podStartSLOduration=1.8982379539999998 podStartE2EDuration="1.898237954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:17:26.897846541 +0000 UTC m=+1528.581955693" watchObservedRunningTime="2026-11-03 02:17:26.898237954 +0000 UTC m=+1528.582347111" Nov 03 02:17:26.905762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:26.905733 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:27.013717 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:27.013636 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24"] Nov 03 02:17:27.018783 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:27.018756 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-4tf24"] Nov 03 02:17:27.130423 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:27.130368 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" podStartSLOduration=2.130353927 podStartE2EDuration="2.130353927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:17:27.127538834 +0000 UTC m=+1528.811648010" watchObservedRunningTime="2026-11-03 02:17:27.130353927 +0000 UTC m=+1528.814463081" Nov 03 02:17:27.247870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:27.247835 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5"] Nov 03 02:17:27.253625 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:27.253584 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-xhbm5"] Nov 03 02:17:27.357362 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:27.357270 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58"] Nov 03 02:17:27.360262 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:27.360233 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-r2x58"] Nov 03 02:17:27.913593 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:27.913548 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:28.445116 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:28.445083 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e5aa4d8-7e07-4efe-96ef-93b35748423e" path="/var/lib/kubelet/pods/1e5aa4d8-7e07-4efe-96ef-93b35748423e/volumes" Nov 03 02:17:28.445667 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:28.445649 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="831449be-effc-4efe-8005-f53a501168a1" path="/var/lib/kubelet/pods/831449be-effc-4efe-8005-f53a501168a1/volumes" Nov 03 02:17:28.446123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:28.446110 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d2c4397-7a8c-41a5-9140-2079f97bd460" path="/var/lib/kubelet/pods/8d2c4397-7a8c-41a5-9140-2079f97bd460/volumes" Nov 03 02:17:28.446916 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:28.446900 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d52f99a6-11e1-4a01-b565-4d3730765a20" path="/var/lib/kubelet/pods/d52f99a6-11e1-4a01-b565-4d3730765a20/volumes" Nov 03 02:17:28.706772 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:28.706692 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:28.707840 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:28.707816 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:17:29.900044 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:29.900015 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:30.065117 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:30.065087 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:30.224169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:30.224098 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:30.432143 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:30.432088 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:31.838593 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:31.838545 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:32.454343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:32.454303 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:33.128997 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.128959 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:33.313503 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.313474 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:33.531716 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.531675 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf"] Nov 03 02:17:33.532646 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.532627 5463 cpu_manager.go:414] "containerMap: removing container" podUID="831449be-effc-4efe-8005-f53a501168a1" containerName="istio-proxy" Nov 03 02:17:33.532700 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.532649 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="831449be-effc-4efe-8005-f53a501168a1" containerName="istio-proxy" Nov 03 02:17:33.532700 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.532671 5463 cpu_manager.go:414] "containerMap: removing container" podUID="831449be-effc-4efe-8005-f53a501168a1" containerName="istio-proxy" Nov 03 02:17:33.532700 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.532678 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="831449be-effc-4efe-8005-f53a501168a1" containerName="istio-proxy" Nov 03 02:17:33.532825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.532739 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d52f99a6-11e1-4a01-b565-4d3730765a20" containerName="istio-proxy" Nov 03 02:17:33.532825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.532747 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52f99a6-11e1-4a01-b565-4d3730765a20" containerName="istio-proxy" Nov 03 02:17:33.532910 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.532898 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="831449be-effc-4efe-8005-f53a501168a1" containerName="istio-proxy" Nov 03 02:17:33.532953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.532915 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="831449be-effc-4efe-8005-f53a501168a1" containerName="istio-proxy" Nov 03 02:17:33.532953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.532929 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52f99a6-11e1-4a01-b565-4d3730765a20" containerName="istio-proxy" Nov 03 02:17:33.552158 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.552133 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf" Nov 03 02:17:33.585092 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.585062 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf"] Nov 03 02:17:33.633797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.632735 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f9b509-dockercfg-7qzxs\"" Nov 03 02:17:33.633797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.632916 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f9b509\"" Nov 03 02:17:33.691312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.691264 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x"] Nov 03 02:17:33.697148 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.697113 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x" Nov 03 02:17:33.712296 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.712264 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf6zw\" (UniqueName: \"kubernetes.io/projected/82305fbb-af59-4971-bb94-137cff38d00b-kube-api-access-vf6zw\") pod \"payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf\" (UID: \"82305fbb-af59-4971-bb94-137cff38d00b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf" Nov 03 02:17:33.712666 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.712642 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82305fbb-af59-4971-bb94-137cff38d00b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf\" (UID: \"82305fbb-af59-4971-bb94-137cff38d00b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf" Nov 03 02:17:33.742400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.742350 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x"] Nov 03 02:17:33.814034 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.813951 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db83203f-b4b2-467f-9fb7-b105d5e89751-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x\" (UID: \"db83203f-b4b2-467f-9fb7-b105d5e89751\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x" Nov 03 02:17:33.814034 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.814007 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf6zw\" (UniqueName: \"kubernetes.io/projected/82305fbb-af59-4971-bb94-137cff38d00b-kube-api-access-vf6zw\") pod \"payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf\" (UID: \"82305fbb-af59-4971-bb94-137cff38d00b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf" Nov 03 02:17:33.814252 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.814231 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zszr6\" (UniqueName: \"kubernetes.io/projected/db83203f-b4b2-467f-9fb7-b105d5e89751-kube-api-access-zszr6\") pod \"payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x\" (UID: \"db83203f-b4b2-467f-9fb7-b105d5e89751\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x" Nov 03 02:17:33.814409 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.814389 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82305fbb-af59-4971-bb94-137cff38d00b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf\" (UID: \"82305fbb-af59-4971-bb94-137cff38d00b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf" Nov 03 02:17:33.815039 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.815021 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82305fbb-af59-4971-bb94-137cff38d00b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf\" (UID: \"82305fbb-af59-4971-bb94-137cff38d00b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf" Nov 03 02:17:33.871027 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.871006 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf6zw\" (UniqueName: \"kubernetes.io/projected/82305fbb-af59-4971-bb94-137cff38d00b-kube-api-access-vf6zw\") pod \"payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf\" (UID: \"82305fbb-af59-4971-bb94-137cff38d00b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf" Nov 03 02:17:33.915713 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.915689 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db83203f-b4b2-467f-9fb7-b105d5e89751-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x\" (UID: \"db83203f-b4b2-467f-9fb7-b105d5e89751\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x" Nov 03 02:17:33.915849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.915786 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zszr6\" (UniqueName: \"kubernetes.io/projected/db83203f-b4b2-467f-9fb7-b105d5e89751-kube-api-access-zszr6\") pod \"payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x\" (UID: \"db83203f-b4b2-467f-9fb7-b105d5e89751\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x" Nov 03 02:17:33.916387 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.916369 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db83203f-b4b2-467f-9fb7-b105d5e89751-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x\" (UID: \"db83203f-b4b2-467f-9fb7-b105d5e89751\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x" Nov 03 02:17:33.972874 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:33.972841 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zszr6\" (UniqueName: \"kubernetes.io/projected/db83203f-b4b2-467f-9fb7-b105d5e89751-kube-api-access-zszr6\") pod \"payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x\" (UID: \"db83203f-b4b2-467f-9fb7-b105d5e89751\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x" Nov 03 02:17:34.016950 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:34.016921 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x" Nov 03 02:17:34.166628 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:34.166599 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf" Nov 03 02:17:34.391146 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:17:34.391111 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb83203f_b4b2_467f_9fb7_b105d5e89751.slice/crio-3c676124f3298c3452bec6f62a287861ba66645e3588950ac02f8355e16403ac WatchSource:0}: Error finding container 3c676124f3298c3452bec6f62a287861ba66645e3588950ac02f8355e16403ac: Status 404 returned error can't find the container with id 3c676124f3298c3452bec6f62a287861ba66645e3588950ac02f8355e16403ac Nov 03 02:17:34.393857 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:34.393827 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x"] Nov 03 02:17:34.545225 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:17:34.545173 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82305fbb_af59_4971_bb94_137cff38d00b.slice/crio-48176717ea5f0f342b794b0f4b30e1c0ecb61e8068f766293f26a56f9fdacfbe WatchSource:0}: Error finding container 48176717ea5f0f342b794b0f4b30e1c0ecb61e8068f766293f26a56f9fdacfbe: Status 404 returned error can't find the container with id 48176717ea5f0f342b794b0f4b30e1c0ecb61e8068f766293f26a56f9fdacfbe Nov 03 02:17:34.548939 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:34.548917 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf"] Nov 03 02:17:34.781508 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:34.781465 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x" event={"ID":"db83203f-b4b2-467f-9fb7-b105d5e89751","Type":"ContainerStarted","Data":"490121feb5482e557715f4df6bcd4c78f83135439d51a419166e1490406fd625"} Nov 03 02:17:34.781721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:34.781542 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x" Nov 03 02:17:34.781721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:34.781568 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x" event={"ID":"db83203f-b4b2-467f-9fb7-b105d5e89751","Type":"ContainerStarted","Data":"3c676124f3298c3452bec6f62a287861ba66645e3588950ac02f8355e16403ac"} Nov 03 02:17:34.783548 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:34.783518 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf" event={"ID":"82305fbb-af59-4971-bb94-137cff38d00b","Type":"ContainerStarted","Data":"b9bacf70db550e4eb4dbd260269a15b28baab037ca29e0d3fb0f311a4313e9bd"} Nov 03 02:17:34.783692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:34.783575 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf" event={"ID":"82305fbb-af59-4971-bb94-137cff38d00b","Type":"ContainerStarted","Data":"48176717ea5f0f342b794b0f4b30e1c0ecb61e8068f766293f26a56f9fdacfbe"} Nov 03 02:17:34.783692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:34.783602 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf" Nov 03 02:17:34.895885 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:34.895834 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x" podStartSLOduration=1.8958213800000001 podStartE2EDuration="1.89582138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:17:34.894453167 +0000 UTC m=+1536.578562323" watchObservedRunningTime="2026-11-03 02:17:34.89582138 +0000 UTC m=+1536.579930534" Nov 03 02:17:35.010847 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:35.010766 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf" podStartSLOduration=2.010746782 podStartE2EDuration="2.010746782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:17:35.008723253 +0000 UTC m=+1536.692832406" watchObservedRunningTime="2026-11-03 02:17:35.010746782 +0000 UTC m=+1536.694855939" Nov 03 02:17:36.836378 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:36.836346 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:37.695589 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:37.695561 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p" Nov 03 02:17:37.695782 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:37.695625 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf" Nov 03 02:17:38.765833 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:38.765795 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:39.519033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:39.519003 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:39.680930 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:39.680898 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:41.088246 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:41.088209 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:41.566030 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:41.565994 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m"] Nov 03 02:17:41.566268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:41.566239 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" podUID="92398fe7-87d2-4c4f-8108-31a05b23c521" containerName="istio-proxy" containerID="cri-o://9e85bf21ced6c467cbfe0f593b74a7b2e15919e62dcb61fa453855f6b49aaa0b" gracePeriod=30 Nov 03 02:17:41.587085 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:41.587058 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:42.399671 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:42.399639 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:43.013808 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:17:43.013770 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:17:43.834146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:43.834109 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:44.363104 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:44.363057 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:44.525251 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:44.525214 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:45.794440 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:45.794410 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x" Nov 03 02:17:45.794946 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:45.794513 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf" Nov 03 02:17:45.874074 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:45.874042 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:46.367114 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.367078 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:46.867626 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.867600 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:17:46.891084 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.891048 5463 generic.go:356] "Generic (PLEG): container finished" podID="92398fe7-87d2-4c4f-8108-31a05b23c521" containerID="9e85bf21ced6c467cbfe0f593b74a7b2e15919e62dcb61fa453855f6b49aaa0b" exitCode=0 Nov 03 02:17:46.891275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.891087 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" Nov 03 02:17:46.891275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.891114 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" event={"ID":"92398fe7-87d2-4c4f-8108-31a05b23c521","Type":"ContainerDied","Data":"9e85bf21ced6c467cbfe0f593b74a7b2e15919e62dcb61fa453855f6b49aaa0b"} Nov 03 02:17:46.891275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.891179 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m" event={"ID":"92398fe7-87d2-4c4f-8108-31a05b23c521","Type":"ContainerDied","Data":"6d5e8a541b64812415a74b2e94652be417e1bb96046417aba043d74a665c7ead"} Nov 03 02:17:46.891275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.891235 5463 scope.go:122] "RemoveContainer" containerID="9e85bf21ced6c467cbfe0f593b74a7b2e15919e62dcb61fa453855f6b49aaa0b" Nov 03 02:17:46.901961 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.901937 5463 scope.go:122] "RemoveContainer" containerID="346e364d7f86c01a662ed57bcf73243a146e9f259580707498be7c99a366ee8a" Nov 03 02:17:46.911591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.911571 5463 scope.go:122] "RemoveContainer" containerID="9e85bf21ced6c467cbfe0f593b74a7b2e15919e62dcb61fa453855f6b49aaa0b" Nov 03 02:17:46.911932 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:17:46.911910 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e85bf21ced6c467cbfe0f593b74a7b2e15919e62dcb61fa453855f6b49aaa0b\": container with ID starting with 9e85bf21ced6c467cbfe0f593b74a7b2e15919e62dcb61fa453855f6b49aaa0b not found: ID does not exist" containerID="9e85bf21ced6c467cbfe0f593b74a7b2e15919e62dcb61fa453855f6b49aaa0b" Nov 03 02:17:46.911988 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.911940 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e85bf21ced6c467cbfe0f593b74a7b2e15919e62dcb61fa453855f6b49aaa0b"} err="failed to get container status \"9e85bf21ced6c467cbfe0f593b74a7b2e15919e62dcb61fa453855f6b49aaa0b\": rpc error: code = NotFound desc = could not find container \"9e85bf21ced6c467cbfe0f593b74a7b2e15919e62dcb61fa453855f6b49aaa0b\": container with ID starting with 9e85bf21ced6c467cbfe0f593b74a7b2e15919e62dcb61fa453855f6b49aaa0b not found: ID does not exist" Nov 03 02:17:46.911988 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.911959 5463 scope.go:122] "RemoveContainer" containerID="346e364d7f86c01a662ed57bcf73243a146e9f259580707498be7c99a366ee8a" Nov 03 02:17:46.912285 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:17:46.912257 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"346e364d7f86c01a662ed57bcf73243a146e9f259580707498be7c99a366ee8a\": container with ID starting with 346e364d7f86c01a662ed57bcf73243a146e9f259580707498be7c99a366ee8a not found: ID does not exist" containerID="346e364d7f86c01a662ed57bcf73243a146e9f259580707498be7c99a366ee8a" Nov 03 02:17:46.912346 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.912296 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"346e364d7f86c01a662ed57bcf73243a146e9f259580707498be7c99a366ee8a"} err="failed to get container status \"346e364d7f86c01a662ed57bcf73243a146e9f259580707498be7c99a366ee8a\": rpc error: code = NotFound desc = could not find container \"346e364d7f86c01a662ed57bcf73243a146e9f259580707498be7c99a366ee8a\": container with ID starting with 346e364d7f86c01a662ed57bcf73243a146e9f259580707498be7c99a366ee8a not found: ID does not exist" Nov 03 02:17:46.922474 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.922451 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-credential-socket\") pod \"92398fe7-87d2-4c4f-8108-31a05b23c521\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " Nov 03 02:17:46.922722 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.922526 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-data\") pod \"92398fe7-87d2-4c4f-8108-31a05b23c521\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " Nov 03 02:17:46.922722 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.922554 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92398fe7-87d2-4c4f-8108-31a05b23c521-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92398fe7-87d2-4c4f-8108-31a05b23c521-istiod-ca-cert\") pod \"92398fe7-87d2-4c4f-8108-31a05b23c521\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " Nov 03 02:17:46.922722 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.922677 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-workload-certs\") pod \"92398fe7-87d2-4c4f-8108-31a05b23c521\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " Nov 03 02:17:46.922722 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.922712 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92398fe7-87d2-4c4f-8108-31a05b23c521-kube-api-access-ztgz8\" (UniqueName: \"kubernetes.io/projected/92398fe7-87d2-4c4f-8108-31a05b23c521-kube-api-access-ztgz8\") pod \"92398fe7-87d2-4c4f-8108-31a05b23c521\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " Nov 03 02:17:46.922993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.922758 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-credential-socket" pod "92398fe7-87d2-4c4f-8108-31a05b23c521" (UID: "92398fe7-87d2-4c4f-8108-31a05b23c521"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:46.922993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.922769 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-token\" (UniqueName: \"kubernetes.io/projected/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-token\") pod \"92398fe7-87d2-4c4f-8108-31a05b23c521\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " Nov 03 02:17:46.922993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.922865 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-data" pod "92398fe7-87d2-4c4f-8108-31a05b23c521" (UID: "92398fe7-87d2-4c4f-8108-31a05b23c521"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:46.922993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.922901 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-workload-socket\") pod \"92398fe7-87d2-4c4f-8108-31a05b23c521\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " Nov 03 02:17:46.922993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.922926 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-envoy\") pod \"92398fe7-87d2-4c4f-8108-31a05b23c521\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " Nov 03 02:17:46.922993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.922953 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-podinfo\") pod \"92398fe7-87d2-4c4f-8108-31a05b23c521\" (UID: \"92398fe7-87d2-4c4f-8108-31a05b23c521\") " Nov 03 02:17:46.923298 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.923121 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-workload-certs" pod "92398fe7-87d2-4c4f-8108-31a05b23c521" (UID: "92398fe7-87d2-4c4f-8108-31a05b23c521"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:46.923298 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.923137 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-workload-socket" pod "92398fe7-87d2-4c4f-8108-31a05b23c521" (UID: "92398fe7-87d2-4c4f-8108-31a05b23c521"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:46.923603 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.923575 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:46.923691 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.923608 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:46.923691 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.923622 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:46.923691 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.923635 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:46.923851 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.923790 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92398fe7-87d2-4c4f-8108-31a05b23c521-istiod-ca-cert" pod "92398fe7-87d2-4c4f-8108-31a05b23c521" (UID: "92398fe7-87d2-4c4f-8108-31a05b23c521"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:17:46.925407 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.925372 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-envoy" pod "92398fe7-87d2-4c4f-8108-31a05b23c521" (UID: "92398fe7-87d2-4c4f-8108-31a05b23c521"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:46.926940 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.926906 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-podinfo" pod "92398fe7-87d2-4c4f-8108-31a05b23c521" (UID: "92398fe7-87d2-4c4f-8108-31a05b23c521"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:17:46.927271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.927245 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92398fe7-87d2-4c4f-8108-31a05b23c521-kube-api-access-ztgz8" pod "92398fe7-87d2-4c4f-8108-31a05b23c521" (UID: "92398fe7-87d2-4c4f-8108-31a05b23c521"). InnerVolumeSpecName "kube-api-access-ztgz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:46.927372 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:46.927312 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-token" pod "92398fe7-87d2-4c4f-8108-31a05b23c521" (UID: "92398fe7-87d2-4c4f-8108-31a05b23c521"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:47.024948 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:47.024910 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92398fe7-87d2-4c4f-8108-31a05b23c521-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:47.024948 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:47.024941 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztgz8\" (UniqueName: \"kubernetes.io/projected/92398fe7-87d2-4c4f-8108-31a05b23c521-kube-api-access-ztgz8\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:47.024948 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:47.024951 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:47.025166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:47.024959 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:47.025166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:47.024968 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92398fe7-87d2-4c4f-8108-31a05b23c521-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:47.315506 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:47.315467 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m"] Nov 03 02:17:47.319080 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:47.319054 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a772c3-openshift-default-69c56d8dc-hhp7m"] Nov 03 02:17:48.444773 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:48.444740 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92398fe7-87d2-4c4f-8108-31a05b23c521" path="/var/lib/kubelet/pods/92398fe7-87d2-4c4f-8108-31a05b23c521/volumes" Nov 03 02:17:48.469683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:48.469641 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:49.074753 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:49.074721 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:49.394240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:49.394203 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6"] Nov 03 02:17:49.501972 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:49.501882 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488"] Nov 03 02:17:49.796447 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:49.796335 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:49.996845 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:49.996770 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:51.489086 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:51.489011 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:51.999919 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:51.999884 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk"] Nov 03 02:17:52.000916 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.000890 5463 cpu_manager.go:414] "containerMap: removing container" podUID="92398fe7-87d2-4c4f-8108-31a05b23c521" containerName="istio-proxy" Nov 03 02:17:52.000994 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.000920 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92398fe7-87d2-4c4f-8108-31a05b23c521" containerName="istio-proxy" Nov 03 02:17:52.001158 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.001146 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="92398fe7-87d2-4c4f-8108-31a05b23c521" containerName="istio-proxy" Nov 03 02:17:52.005603 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.005580 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.058324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.058295 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk"] Nov 03 02:17:52.097183 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.097154 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:52.121337 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.121307 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6d496ad-cf03-4990-9326-a127a83750e2-istio-token\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.121493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.121379 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c725\" (UniqueName: \"kubernetes.io/projected/a6d496ad-cf03-4990-9326-a127a83750e2-kube-api-access-6c725\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.121493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.121427 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-workload-socket\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.121721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.121704 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-credential-socket\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.121755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.121734 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-istio-envoy\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.121904 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.121879 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-workload-certs\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.121946 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.121915 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-istio-data\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.122035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.122006 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6d496ad-cf03-4990-9326-a127a83750e2-istiod-ca-cert\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.122077 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.122063 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6d496ad-cf03-4990-9326-a127a83750e2-istio-podinfo\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.152045 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.152009 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ff26f5-openshift-default-dockercfg-s6w2d\"" Nov 03 02:17:52.227208 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.227125 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-credential-socket\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.227208 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.227174 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-istio-envoy\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.227457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.227271 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-workload-certs\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.227457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.227299 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-istio-data\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.227457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.227343 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6d496ad-cf03-4990-9326-a127a83750e2-istiod-ca-cert\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.227457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.227382 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6d496ad-cf03-4990-9326-a127a83750e2-istio-podinfo\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.227457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.227436 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6d496ad-cf03-4990-9326-a127a83750e2-istio-token\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.227743 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.227490 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c725\" (UniqueName: \"kubernetes.io/projected/a6d496ad-cf03-4990-9326-a127a83750e2-kube-api-access-6c725\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.227743 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.227535 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-workload-socket\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.228043 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.228016 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-workload-socket\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.228283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.228256 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-workload-certs\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.228467 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.228332 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-credential-socket\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.228591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.228377 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-istio-data\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.269268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.263431 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6d496ad-cf03-4990-9326-a127a83750e2-istiod-ca-cert\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.269449 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.269415 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-istio-envoy\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.275240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.269897 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6d496ad-cf03-4990-9326-a127a83750e2-istio-podinfo\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.292152 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.292123 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c725\" (UniqueName: \"kubernetes.io/projected/a6d496ad-cf03-4990-9326-a127a83750e2-kube-api-access-6c725\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.292152 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.292137 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6d496ad-cf03-4990-9326-a127a83750e2-istio-token\") pod \"e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.322055 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.322025 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:52.374912 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.374863 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:52.756233 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:17:52.756201 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d496ad_cf03_4990_9326_a127a83750e2.slice/crio-d6c25f3b9ec4b970a6e00d268c9c0481634d6e3be4ec08c4054676978cc10bd3 WatchSource:0}: Error finding container d6c25f3b9ec4b970a6e00d268c9c0481634d6e3be4ec08c4054676978cc10bd3: Status 404 returned error can't find the container with id d6c25f3b9ec4b970a6e00d268c9c0481634d6e3be4ec08c4054676978cc10bd3 Nov 03 02:17:52.758703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.758666 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:17:52.758803 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.758736 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:17:52.758803 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.758768 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:17:52.759726 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.759706 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk"] Nov 03 02:17:52.964755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.964716 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" event={"ID":"a6d496ad-cf03-4990-9326-a127a83750e2","Type":"ContainerStarted","Data":"2d5583ec3f7631bc8c0ff3725312d77683e20eacfffc051fcba9943bb2b725c2"} Nov 03 02:17:52.964948 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:52.964815 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" event={"ID":"a6d496ad-cf03-4990-9326-a127a83750e2","Type":"ContainerStarted","Data":"d6c25f3b9ec4b970a6e00d268c9c0481634d6e3be4ec08c4054676978cc10bd3"} Nov 03 02:17:53.085029 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:53.084877 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" podStartSLOduration=2.084861869 podStartE2EDuration="2.084861869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:17:53.082084576 +0000 UTC m=+1554.766193732" watchObservedRunningTime="2026-11-03 02:17:53.084861869 +0000 UTC m=+1554.768971023" Nov 03 02:17:53.322452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:53.322420 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:53.327794 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:53.327772 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:53.872969 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:53.872933 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2"] Nov 03 02:17:53.873807 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:53.873784 5463 cpu_manager.go:414] "containerMap: removing container" podUID="92398fe7-87d2-4c4f-8108-31a05b23c521" containerName="istio-proxy" Nov 03 02:17:53.873915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:53.873808 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92398fe7-87d2-4c4f-8108-31a05b23c521" containerName="istio-proxy" Nov 03 02:17:53.874008 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:53.873993 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="92398fe7-87d2-4c4f-8108-31a05b23c521" containerName="istio-proxy" Nov 03 02:17:53.877821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:53.877793 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" Nov 03 02:17:53.930753 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:53.930723 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-701ba30f-serving-cert\"" Nov 03 02:17:53.930753 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:53.930727 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:17:53.930964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:53.930783 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:17:53.930964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:53.930830 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qv59l\"" Nov 03 02:17:53.931050 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:53.931003 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2"] Nov 03 02:17:53.950883 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:53.950855 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgj5w\" (UniqueName: \"kubernetes.io/projected/698b1c2c-aed3-496c-9a2d-dbe3137090f6-kube-api-access-xgj5w\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2\" (UID: \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" Nov 03 02:17:53.951042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:53.950917 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/698b1c2c-aed3-496c-9a2d-dbe3137090f6-maas-api-tls\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2\" (UID: \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" Nov 03 02:17:53.951042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:53.950996 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/698b1c2c-aed3-496c-9a2d-dbe3137090f6-openshift-service-ca\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2\" (UID: \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" Nov 03 02:17:53.951042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:53.951021 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698b1c2c-aed3-496c-9a2d-dbe3137090f6-metrics-certs\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2\" (UID: \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" Nov 03 02:17:53.974818 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:53.974548 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:53.975373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:53.975353 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:17:53.977692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:53.977672 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 02:17:54.052010 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.051972 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/698b1c2c-aed3-496c-9a2d-dbe3137090f6-openshift-service-ca\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2\" (UID: \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" Nov 03 02:17:54.052010 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.052006 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698b1c2c-aed3-496c-9a2d-dbe3137090f6-metrics-certs\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2\" (UID: \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" Nov 03 02:17:54.052297 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.052220 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgj5w\" (UniqueName: \"kubernetes.io/projected/698b1c2c-aed3-496c-9a2d-dbe3137090f6-kube-api-access-xgj5w\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2\" (UID: \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" Nov 03 02:17:54.052297 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.052289 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/698b1c2c-aed3-496c-9a2d-dbe3137090f6-maas-api-tls\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2\" (UID: \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" Nov 03 02:17:54.052668 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:17:54.052430 5463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-serving-cert: secret "maas-api-e2e-mt-701ba30f-serving-cert" not found Nov 03 02:17:54.052668 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:17:54.052509 5463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/698b1c2c-aed3-496c-9a2d-dbe3137090f6-maas-api-tls podName:698b1c2c-aed3-496c-9a2d-dbe3137090f6 nodeName:}" failed. No retries permitted until 2026-11-03 02:17:54.552490781 +0000 UTC m=+1556.236599913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/698b1c2c-aed3-496c-9a2d-dbe3137090f6-maas-api-tls") pod "maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" (UID: "698b1c2c-aed3-496c-9a2d-dbe3137090f6") : secret "maas-api-e2e-mt-701ba30f-serving-cert" not found Nov 03 02:17:54.052822 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.052788 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/698b1c2c-aed3-496c-9a2d-dbe3137090f6-openshift-service-ca\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2\" (UID: \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" Nov 03 02:17:54.054675 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.054651 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698b1c2c-aed3-496c-9a2d-dbe3137090f6-metrics-certs\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2\" (UID: \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" Nov 03 02:17:54.094044 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.094016 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:54.112001 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.111973 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgj5w\" (UniqueName: \"kubernetes.io/projected/698b1c2c-aed3-496c-9a2d-dbe3137090f6-kube-api-access-xgj5w\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2\" (UID: \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" Nov 03 02:17:54.259673 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.259644 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:54.394880 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.394838 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6" podUID="c1f85293-820a-436c-a860-55b068a4482a" containerName="payload-processing" containerID="cri-o://2933f0c0b1b710f237dc9d575f7e7f5a9608488660872177141855185bf04c30" gracePeriod=25 Nov 03 02:17:54.418907 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.418883 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:54.503276 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.503230 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488" podUID="1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0" containerName="payload-pre-processing" containerID="cri-o://39af84c07019a1095a44c4266e6ec6e5c76eec98d341280077f7edf08fb3b57d" gracePeriod=25 Nov 03 02:17:54.558923 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.558885 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/698b1c2c-aed3-496c-9a2d-dbe3137090f6-maas-api-tls\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2\" (UID: \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" Nov 03 02:17:54.561661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.561631 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/698b1c2c-aed3-496c-9a2d-dbe3137090f6-maas-api-tls\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2\" (UID: \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" Nov 03 02:17:54.564417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.564393 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:54.718231 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.710585 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6" Nov 03 02:17:54.763406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.762342 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1f85293-820a-436c-a860-55b068a4482a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f85293-820a-436c-a860-55b068a4482a-plugins-config-volume\") pod \"c1f85293-820a-436c-a860-55b068a4482a\" (UID: \"c1f85293-820a-436c-a860-55b068a4482a\") " Nov 03 02:17:54.763406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.762399 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f85293-820a-436c-a860-55b068a4482a-kube-api-access-b8gkl\" (UniqueName: \"kubernetes.io/projected/c1f85293-820a-436c-a860-55b068a4482a-kube-api-access-b8gkl\") pod \"c1f85293-820a-436c-a860-55b068a4482a\" (UID: \"c1f85293-820a-436c-a860-55b068a4482a\") " Nov 03 02:17:54.763406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.762731 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f85293-820a-436c-a860-55b068a4482a-plugins-config-volume" pod "c1f85293-820a-436c-a860-55b068a4482a" (UID: "c1f85293-820a-436c-a860-55b068a4482a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:17:54.771474 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.763878 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f85293-820a-436c-a860-55b068a4482a-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:54.771474 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.765945 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f85293-820a-436c-a860-55b068a4482a-kube-api-access-b8gkl" pod "c1f85293-820a-436c-a860-55b068a4482a" (UID: "c1f85293-820a-436c-a860-55b068a4482a"). InnerVolumeSpecName "kube-api-access-b8gkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:54.794229 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.794200 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" Nov 03 02:17:54.799933 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.799912 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488" Nov 03 02:17:54.868441 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.864540 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0-plugins-config-volume\") pod \"1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0\" (UID: \"1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0\") " Nov 03 02:17:54.868441 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.864673 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0-kube-api-access-krbhb\" (UniqueName: \"kubernetes.io/projected/1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0-kube-api-access-krbhb\") pod \"1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0\" (UID: \"1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0\") " Nov 03 02:17:54.868441 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.865141 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8gkl\" (UniqueName: \"kubernetes.io/projected/c1f85293-820a-436c-a860-55b068a4482a-kube-api-access-b8gkl\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:54.868441 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.865794 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0-plugins-config-volume" pod "1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0" (UID: "1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:17:54.868441 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.867442 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0-kube-api-access-krbhb" pod "1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0" (UID: "1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0"). InnerVolumeSpecName "kube-api-access-krbhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:54.966491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.966456 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:54.966491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.966485 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krbhb\" (UniqueName: \"kubernetes.io/projected/1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0-kube-api-access-krbhb\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:54.985241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.985168 5463 generic.go:356] "Generic (PLEG): container finished" podID="c1f85293-820a-436c-a860-55b068a4482a" containerID="2933f0c0b1b710f237dc9d575f7e7f5a9608488660872177141855185bf04c30" exitCode=0 Nov 03 02:17:54.985399 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.985304 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6" event={"ID":"c1f85293-820a-436c-a860-55b068a4482a","Type":"ContainerDied","Data":"2933f0c0b1b710f237dc9d575f7e7f5a9608488660872177141855185bf04c30"} Nov 03 02:17:54.985399 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.985378 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6" event={"ID":"c1f85293-820a-436c-a860-55b068a4482a","Type":"ContainerDied","Data":"f8c64c96dde0a86856516af48e9cb04817035c2ea6b9fe9f8cca6d6e97ddba66"} Nov 03 02:17:54.985505 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.985407 5463 scope.go:122] "RemoveContainer" containerID="2933f0c0b1b710f237dc9d575f7e7f5a9608488660872177141855185bf04c30" Nov 03 02:17:54.985562 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.985546 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6" Nov 03 02:17:54.990433 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.988279 5463 generic.go:356] "Generic (PLEG): container finished" podID="1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0" containerID="39af84c07019a1095a44c4266e6ec6e5c76eec98d341280077f7edf08fb3b57d" exitCode=0 Nov 03 02:17:54.990433 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.989621 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488" Nov 03 02:17:54.995412 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.995346 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488" event={"ID":"1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0","Type":"ContainerDied","Data":"39af84c07019a1095a44c4266e6ec6e5c76eec98d341280077f7edf08fb3b57d"} Nov 03 02:17:54.995412 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:54.995399 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488" event={"ID":"1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0","Type":"ContainerDied","Data":"e84e8cf0a6f04df3393f5f47b5686f0d77c037a0bb79894570c63c0cd043f696"} Nov 03 02:17:55.003620 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:55.003594 5463 scope.go:122] "RemoveContainer" containerID="2933f0c0b1b710f237dc9d575f7e7f5a9608488660872177141855185bf04c30" Nov 03 02:17:55.003920 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:17:55.003887 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2933f0c0b1b710f237dc9d575f7e7f5a9608488660872177141855185bf04c30\": container with ID starting with 2933f0c0b1b710f237dc9d575f7e7f5a9608488660872177141855185bf04c30 not found: ID does not exist" containerID="2933f0c0b1b710f237dc9d575f7e7f5a9608488660872177141855185bf04c30" Nov 03 02:17:55.004002 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:55.003927 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2933f0c0b1b710f237dc9d575f7e7f5a9608488660872177141855185bf04c30"} err="failed to get container status \"2933f0c0b1b710f237dc9d575f7e7f5a9608488660872177141855185bf04c30\": rpc error: code = NotFound desc = could not find container \"2933f0c0b1b710f237dc9d575f7e7f5a9608488660872177141855185bf04c30\": container with ID starting with 2933f0c0b1b710f237dc9d575f7e7f5a9608488660872177141855185bf04c30 not found: ID does not exist" Nov 03 02:17:55.004002 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:55.003952 5463 scope.go:122] "RemoveContainer" containerID="39af84c07019a1095a44c4266e6ec6e5c76eec98d341280077f7edf08fb3b57d" Nov 03 02:17:55.013003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:55.012985 5463 scope.go:122] "RemoveContainer" containerID="39af84c07019a1095a44c4266e6ec6e5c76eec98d341280077f7edf08fb3b57d" Nov 03 02:17:55.013278 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:17:55.013261 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39af84c07019a1095a44c4266e6ec6e5c76eec98d341280077f7edf08fb3b57d\": container with ID starting with 39af84c07019a1095a44c4266e6ec6e5c76eec98d341280077f7edf08fb3b57d not found: ID does not exist" containerID="39af84c07019a1095a44c4266e6ec6e5c76eec98d341280077f7edf08fb3b57d" Nov 03 02:17:55.013323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:55.013286 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39af84c07019a1095a44c4266e6ec6e5c76eec98d341280077f7edf08fb3b57d"} err="failed to get container status \"39af84c07019a1095a44c4266e6ec6e5c76eec98d341280077f7edf08fb3b57d\": rpc error: code = NotFound desc = could not find container \"39af84c07019a1095a44c4266e6ec6e5c76eec98d341280077f7edf08fb3b57d\": container with ID starting with 39af84c07019a1095a44c4266e6ec6e5c76eec98d341280077f7edf08fb3b57d not found: ID does not exist" Nov 03 02:17:55.154898 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:55.154866 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488"] Nov 03 02:17:55.159847 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:55.159821 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aa82d2-8675df8fc5-4q488"] Nov 03 02:17:55.185548 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:17:55.185517 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698b1c2c_aed3_496c_9a2d_dbe3137090f6.slice/crio-18e708b0d85c6f7f456bf4d5ad005b63e5d17cf82604981bcdf030f0a469edd2 WatchSource:0}: Error finding container 18e708b0d85c6f7f456bf4d5ad005b63e5d17cf82604981bcdf030f0a469edd2: Status 404 returned error can't find the container with id 18e708b0d85c6f7f456bf4d5ad005b63e5d17cf82604981bcdf030f0a469edd2 Nov 03 02:17:55.187378 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:55.187355 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2"] Nov 03 02:17:55.187725 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:55.187701 5463 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 02:17:55.268713 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:55.268685 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6"] Nov 03 02:17:55.272801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:55.272774 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-aa82d2-6f847f7f7-lxxz6"] Nov 03 02:17:55.905087 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:55.905050 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:56.001556 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:56.001522 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" event={"ID":"698b1c2c-aed3-496c-9a2d-dbe3137090f6","Type":"ContainerStarted","Data":"18e708b0d85c6f7f456bf4d5ad005b63e5d17cf82604981bcdf030f0a469edd2"} Nov 03 02:17:56.088288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:56.088218 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:56.255065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:56.254851 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:56.443649 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:56.443618 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0" path="/var/lib/kubelet/pods/1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0/volumes" Nov 03 02:17:56.444053 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:56.444034 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1f85293-820a-436c-a860-55b068a4482a" path="/var/lib/kubelet/pods/c1f85293-820a-436c-a860-55b068a4482a/volumes" Nov 03 02:17:59.149710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.149662 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:59.343387 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.343337 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:59.546812 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.546700 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9"] Nov 03 02:17:59.547718 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.547693 5463 cpu_manager.go:414] "containerMap: removing container" podUID="1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0" containerName="payload-pre-processing" Nov 03 02:17:59.547845 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.547719 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0" containerName="payload-pre-processing" Nov 03 02:17:59.547845 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.547783 5463 cpu_manager.go:414] "containerMap: removing container" podUID="c1f85293-820a-436c-a860-55b068a4482a" containerName="payload-processing" Nov 03 02:17:59.547845 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.547792 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f85293-820a-436c-a860-55b068a4482a" containerName="payload-processing" Nov 03 02:17:59.547990 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.547958 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f85293-820a-436c-a860-55b068a4482a" containerName="payload-processing" Nov 03 02:17:59.548087 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.548038 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef23d0c-f08f-4bf8-9268-3ea93e6bd7b0" containerName="payload-pre-processing" Nov 03 02:17:59.553716 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.553694 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9" Nov 03 02:17:59.601625 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.601594 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9"] Nov 03 02:17:59.640510 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.640272 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwqkx\" (UniqueName: \"kubernetes.io/projected/cd144ab9-c755-4886-a81f-cbd747a5e5c3-kube-api-access-zwqkx\") pod \"payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9\" (UID: \"cd144ab9-c755-4886-a81f-cbd747a5e5c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9" Nov 03 02:17:59.640510 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.640348 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd144ab9-c755-4886-a81f-cbd747a5e5c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9\" (UID: \"cd144ab9-c755-4886-a81f-cbd747a5e5c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9" Nov 03 02:17:59.656912 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.649910 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ff26f5\"" Nov 03 02:17:59.656912 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.650014 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ff26f5-dockercfg-rt4vq\"" Nov 03 02:17:59.704220 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.699729 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2"] Nov 03 02:17:59.733243 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.726111 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2" Nov 03 02:17:59.742810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.742435 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwqkx\" (UniqueName: \"kubernetes.io/projected/cd144ab9-c755-4886-a81f-cbd747a5e5c3-kube-api-access-zwqkx\") pod \"payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9\" (UID: \"cd144ab9-c755-4886-a81f-cbd747a5e5c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9" Nov 03 02:17:59.742810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.742508 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd144ab9-c755-4886-a81f-cbd747a5e5c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9\" (UID: \"cd144ab9-c755-4886-a81f-cbd747a5e5c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9" Nov 03 02:17:59.744762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.744238 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd144ab9-c755-4886-a81f-cbd747a5e5c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9\" (UID: \"cd144ab9-c755-4886-a81f-cbd747a5e5c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9" Nov 03 02:17:59.794330 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.794281 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2"] Nov 03 02:17:59.817603 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.817531 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwqkx\" (UniqueName: \"kubernetes.io/projected/cd144ab9-c755-4886-a81f-cbd747a5e5c3-kube-api-access-zwqkx\") pod \"payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9\" (UID: \"cd144ab9-c755-4886-a81f-cbd747a5e5c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9" Nov 03 02:17:59.845527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.845490 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tfq7\" (UniqueName: \"kubernetes.io/projected/8fbbc88e-2111-44d1-a28d-537cc9aa1094-kube-api-access-6tfq7\") pod \"payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2\" (UID: \"8fbbc88e-2111-44d1-a28d-537cc9aa1094\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2" Nov 03 02:17:59.845694 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.845644 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbbc88e-2111-44d1-a28d-537cc9aa1094-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2\" (UID: \"8fbbc88e-2111-44d1-a28d-537cc9aa1094\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2" Nov 03 02:17:59.876937 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.876907 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:59.877415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.877393 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9" Nov 03 02:17:59.950242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.947229 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tfq7\" (UniqueName: \"kubernetes.io/projected/8fbbc88e-2111-44d1-a28d-537cc9aa1094-kube-api-access-6tfq7\") pod \"payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2\" (UID: \"8fbbc88e-2111-44d1-a28d-537cc9aa1094\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2" Nov 03 02:17:59.950242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.947353 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbbc88e-2111-44d1-a28d-537cc9aa1094-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2\" (UID: \"8fbbc88e-2111-44d1-a28d-537cc9aa1094\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2" Nov 03 02:17:59.950242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:17:59.948672 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbbc88e-2111-44d1-a28d-537cc9aa1094-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2\" (UID: \"8fbbc88e-2111-44d1-a28d-537cc9aa1094\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2" Nov 03 02:18:00.005240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:00.005212 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tfq7\" (UniqueName: \"kubernetes.io/projected/8fbbc88e-2111-44d1-a28d-537cc9aa1094-kube-api-access-6tfq7\") pod \"payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2\" (UID: \"8fbbc88e-2111-44d1-a28d-537cc9aa1094\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2" Nov 03 02:18:00.100167 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:00.100089 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2" Nov 03 02:18:00.294638 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:00.294609 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9"] Nov 03 02:18:00.302975 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:18:00.302947 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd144ab9_c755_4886_a81f_cbd747a5e5c3.slice/crio-ab06aaaa149f0a050dfe819a32a5d51cf541cbc099e10d9cccb349ef9cf2ccc4 WatchSource:0}: Error finding container ab06aaaa149f0a050dfe819a32a5d51cf541cbc099e10d9cccb349ef9cf2ccc4: Status 404 returned error can't find the container with id ab06aaaa149f0a050dfe819a32a5d51cf541cbc099e10d9cccb349ef9cf2ccc4 Nov 03 02:18:00.484970 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:18:00.484938 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbbc88e_2111_44d1_a28d_537cc9aa1094.slice/crio-8f8eaf26269eec8c7cabee2d929b80cb91b7a825698b61d30cd61870956ea8e3 WatchSource:0}: Error finding container 8f8eaf26269eec8c7cabee2d929b80cb91b7a825698b61d30cd61870956ea8e3: Status 404 returned error can't find the container with id 8f8eaf26269eec8c7cabee2d929b80cb91b7a825698b61d30cd61870956ea8e3 Nov 03 02:18:00.487875 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:00.487849 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2"] Nov 03 02:18:01.044388 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:01.044354 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" event={"ID":"698b1c2c-aed3-496c-9a2d-dbe3137090f6","Type":"ContainerStarted","Data":"1cc007583eca8acf71f9e35415bc525090e63dbad614a48070435a98f74d8a3f"} Nov 03 02:18:01.044681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:01.044437 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" Nov 03 02:18:01.046020 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:01.046001 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2" event={"ID":"8fbbc88e-2111-44d1-a28d-537cc9aa1094","Type":"ContainerStarted","Data":"eaf3617fcfdddabfa5740500b8149d8aaf0896cc9e34ff6395f79d061d46ef74"} Nov 03 02:18:01.046133 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:01.046036 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2" event={"ID":"8fbbc88e-2111-44d1-a28d-537cc9aa1094","Type":"ContainerStarted","Data":"8f8eaf26269eec8c7cabee2d929b80cb91b7a825698b61d30cd61870956ea8e3"} Nov 03 02:18:01.046133 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:01.046082 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2" Nov 03 02:18:01.047573 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:01.047552 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9" event={"ID":"cd144ab9-c755-4886-a81f-cbd747a5e5c3","Type":"ContainerStarted","Data":"7fb5872e8eaf0cf2079899b3d17f65f51a6061964249621a46e59128c8f1a234"} Nov 03 02:18:01.047708 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:01.047585 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9" event={"ID":"cd144ab9-c755-4886-a81f-cbd747a5e5c3","Type":"ContainerStarted","Data":"ab06aaaa149f0a050dfe819a32a5d51cf541cbc099e10d9cccb349ef9cf2ccc4"} Nov 03 02:18:01.047708 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:01.047671 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9" Nov 03 02:18:01.161168 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:01.161114 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" podStartSLOduration=3.030537243 podStartE2EDuration="8.16109737s" totalImagesPullingTime="5.130560127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:17:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:18:01.158659411 +0000 UTC m=+1562.842768566" watchObservedRunningTime="2026-11-03 02:18:01.16109737 +0000 UTC m=+1562.845206526" Nov 03 02:18:01.290274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:01.288399 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9" podStartSLOduration=2.288377936 podStartE2EDuration="2.288377936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:18:01.286362919 +0000 UTC m=+1562.970472075" watchObservedRunningTime="2026-11-03 02:18:01.288377936 +0000 UTC m=+1562.972487091" Nov 03 02:18:01.400244 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:01.400166 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2" podStartSLOduration=2.400148062 podStartE2EDuration="2.400148062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:18:01.399025794 +0000 UTC m=+1563.083134951" watchObservedRunningTime="2026-11-03 02:18:01.400148062 +0000 UTC m=+1563.084257219" Nov 03 02:18:03.898900 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:03.898870 5463 scope.go:122] "RemoveContainer" containerID="dcdf053bfeba3ea84b2f1099f8f883bb39e4f67de28ccf31d9bc544f0ec9f4e5" Nov 03 02:18:03.908963 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:03.908939 5463 scope.go:122] "RemoveContainer" containerID="4e3f3e8d6220b64067c73f3e35dfa8e8ea9a4e14bd1da21d85dae7379aa7f717" Nov 03 02:18:03.917935 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:03.917915 5463 scope.go:122] "RemoveContainer" containerID="5ddf65b000bd2ba7004d83f0a4f3c08f651fb17a6b6ca68f2f9fb10f57c7b3af" Nov 03 02:18:03.926627 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:03.926602 5463 scope.go:122] "RemoveContainer" containerID="2827fdf0c5c39719805bd689e30d7a12423a75bcb874f5d918a08ea5b18af63e" Nov 03 02:18:03.935063 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:03.935046 5463 scope.go:122] "RemoveContainer" containerID="ebaf64202acc66878448e62ad1da0625d9fde336944f8f5633b4e7e3aa8112b0" Nov 03 02:18:04.725303 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:04.725271 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:07.060046 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:07.060017 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" Nov 03 02:18:09.524172 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:09.524140 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:12.055406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:12.055368 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:12.057802 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:12.057767 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2" Nov 03 02:18:12.058038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:12.058015 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9" Nov 03 02:18:13.007253 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:18:13.007184 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:18:14.230689 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:14.230653 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:15.887640 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:15.887602 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms"] Nov 03 02:18:15.888098 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:15.887800 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" podUID="2394b438-fc9a-4747-8eb8-78e9f1524a1d" containerName="istio-proxy" containerID="cri-o://4199f540700f2ed29d2a34f70e3dec7b0e78872698e6bb48e703bd78a790c940" gracePeriod=30 Nov 03 02:18:15.938411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:15.938380 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv"] Nov 03 02:18:15.938588 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:15.938567 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" podUID="30a0bcec-82be-4efd-bd03-87dadb66b21b" containerName="istio-proxy" containerID="cri-o://4ea19b22cb34eb9d62384865369b7a81ebabed16b32a994dde67b2e4ca656637" gracePeriod=30 Nov 03 02:18:15.941532 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:15.941506 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p"] Nov 03 02:18:15.941705 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:15.941685 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" podUID="547c8528-2b07-475d-a191-f0de8a63a4b9" containerName="istio-proxy" containerID="cri-o://4ff036340ad598d738328bc2046c567f0dde8508f0aa04e6d148a235e8a3bb2c" gracePeriod=30 Nov 03 02:18:18.650099 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:18.650062 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p"] Nov 03 02:18:18.655100 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:18.655079 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" Nov 03 02:18:18.706827 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:18.706795 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p"] Nov 03 02:18:18.753118 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:18.753088 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f9b509-serving-cert\"" Nov 03 02:18:18.782355 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:18.782312 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-metrics-certs\") pod \"maas-api-e2e-api-b-f9b509-7d565bc76-sx45p\" (UID: \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" Nov 03 02:18:18.782563 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:18.782543 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-maas-api-tls\") pod \"maas-api-e2e-api-b-f9b509-7d565bc76-sx45p\" (UID: \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" Nov 03 02:18:18.782635 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:18.782586 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-openshift-service-ca\") pod \"maas-api-e2e-api-b-f9b509-7d565bc76-sx45p\" (UID: \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" Nov 03 02:18:18.782700 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:18.782656 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79qdg\" (UniqueName: \"kubernetes.io/projected/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-kube-api-access-79qdg\") pod \"maas-api-e2e-api-b-f9b509-7d565bc76-sx45p\" (UID: \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" Nov 03 02:18:18.865469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:18.865427 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:18.884731 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:18.884696 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-maas-api-tls\") pod \"maas-api-e2e-api-b-f9b509-7d565bc76-sx45p\" (UID: \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" Nov 03 02:18:18.884923 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:18.884751 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-openshift-service-ca\") pod \"maas-api-e2e-api-b-f9b509-7d565bc76-sx45p\" (UID: \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" Nov 03 02:18:18.884923 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:18.884827 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79qdg\" (UniqueName: \"kubernetes.io/projected/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-kube-api-access-79qdg\") pod \"maas-api-e2e-api-b-f9b509-7d565bc76-sx45p\" (UID: \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" Nov 03 02:18:18.884923 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:18.884900 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-metrics-certs\") pod \"maas-api-e2e-api-b-f9b509-7d565bc76-sx45p\" (UID: \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" Nov 03 02:18:18.885107 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:18:18.884955 5463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-serving-cert: secret "maas-api-e2e-api-b-f9b509-serving-cert" not found Nov 03 02:18:18.885107 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:18:18.885036 5463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-maas-api-tls podName:f1a2b779-6480-46dc-b9d2-05c3ef91bcb7 nodeName:}" failed. No retries permitted until 2026-11-03 02:18:19.385015554 +0000 UTC m=+1581.069124702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-maas-api-tls") pod "maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" (UID: "f1a2b779-6480-46dc-b9d2-05c3ef91bcb7") : secret "maas-api-e2e-api-b-f9b509-serving-cert" not found Nov 03 02:18:18.886079 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:18.886047 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-openshift-service-ca\") pod \"maas-api-e2e-api-b-f9b509-7d565bc76-sx45p\" (UID: \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" Nov 03 02:18:18.887996 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:18.887968 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-metrics-certs\") pod \"maas-api-e2e-api-b-f9b509-7d565bc76-sx45p\" (UID: \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" Nov 03 02:18:18.948287 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:18.943889 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79qdg\" (UniqueName: \"kubernetes.io/projected/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-kube-api-access-79qdg\") pod \"maas-api-e2e-api-b-f9b509-7d565bc76-sx45p\" (UID: \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" Nov 03 02:18:19.071217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:19.070294 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:19.226522 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:19.226437 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:19.392839 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:19.392798 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-maas-api-tls\") pod \"maas-api-e2e-api-b-f9b509-7d565bc76-sx45p\" (UID: \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" Nov 03 02:18:19.395445 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:19.395416 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-maas-api-tls\") pod \"maas-api-e2e-api-b-f9b509-7d565bc76-sx45p\" (UID: \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" Nov 03 02:18:19.573218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:19.573129 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" Nov 03 02:18:19.941995 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:18:19.941943 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a2b779_6480_46dc_b9d2_05c3ef91bcb7.slice/crio-0bd9ba7da4df16493704e0bcdfc44d8bcc63343fca12314a313a6a7baf30ad6e WatchSource:0}: Error finding container 0bd9ba7da4df16493704e0bcdfc44d8bcc63343fca12314a313a6a7baf30ad6e: Status 404 returned error can't find the container with id 0bd9ba7da4df16493704e0bcdfc44d8bcc63343fca12314a313a6a7baf30ad6e Nov 03 02:18:19.946362 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:19.946338 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p"] Nov 03 02:18:20.207321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:20.207230 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" event={"ID":"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7","Type":"ContainerStarted","Data":"0bd9ba7da4df16493704e0bcdfc44d8bcc63343fca12314a313a6a7baf30ad6e"} Nov 03 02:18:21.223403 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.223365 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" event={"ID":"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7","Type":"ContainerStarted","Data":"8942618ae8b97f547144c5fe9900ec6b7879f84a1942e026cca2a949ab2ed405"} Nov 03 02:18:21.223853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.223661 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" Nov 03 02:18:21.225904 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.225793 5463 generic.go:356] "Generic (PLEG): container finished" podID="30a0bcec-82be-4efd-bd03-87dadb66b21b" containerID="4ea19b22cb34eb9d62384865369b7a81ebabed16b32a994dde67b2e4ca656637" exitCode=0 Nov 03 02:18:21.225904 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.225885 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" event={"ID":"30a0bcec-82be-4efd-bd03-87dadb66b21b","Type":"ContainerDied","Data":"4ea19b22cb34eb9d62384865369b7a81ebabed16b32a994dde67b2e4ca656637"} Nov 03 02:18:21.235703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.235667 5463 generic.go:356] "Generic (PLEG): container finished" podID="2394b438-fc9a-4747-8eb8-78e9f1524a1d" containerID="4199f540700f2ed29d2a34f70e3dec7b0e78872698e6bb48e703bd78a790c940" exitCode=0 Nov 03 02:18:21.235802 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.235742 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" event={"ID":"2394b438-fc9a-4747-8eb8-78e9f1524a1d","Type":"ContainerDied","Data":"4199f540700f2ed29d2a34f70e3dec7b0e78872698e6bb48e703bd78a790c940"} Nov 03 02:18:21.235863 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.235818 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" event={"ID":"2394b438-fc9a-4747-8eb8-78e9f1524a1d","Type":"ContainerDied","Data":"a0da7de4be5a16aedc933891fc61a5a572749de08db3ddbc4132761e261b35ec"} Nov 03 02:18:21.235863 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.235845 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0da7de4be5a16aedc933891fc61a5a572749de08db3ddbc4132761e261b35ec" Nov 03 02:18:21.239916 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.237560 5463 generic.go:356] "Generic (PLEG): container finished" podID="547c8528-2b07-475d-a191-f0de8a63a4b9" containerID="4ff036340ad598d738328bc2046c567f0dde8508f0aa04e6d148a235e8a3bb2c" exitCode=0 Nov 03 02:18:21.239916 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.237602 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" event={"ID":"547c8528-2b07-475d-a191-f0de8a63a4b9","Type":"ContainerDied","Data":"4ff036340ad598d738328bc2046c567f0dde8508f0aa04e6d148a235e8a3bb2c"} Nov 03 02:18:21.254169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.254150 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:18:21.302916 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.302896 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:18:21.306773 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.306751 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:18:21.315745 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.315725 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-workload-socket\") pod \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " Nov 03 02:18:21.315849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.315767 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-data\") pod \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " Nov 03 02:18:21.315891 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.315854 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-token\" (UniqueName: \"kubernetes.io/projected/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-token\") pod \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " Nov 03 02:18:21.315934 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.315889 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-workload-certs\") pod \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " Nov 03 02:18:21.315969 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.315947 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-envoy\") pod \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " Nov 03 02:18:21.316023 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.315978 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-workload-socket" pod "2394b438-fc9a-4747-8eb8-78e9f1524a1d" (UID: "2394b438-fc9a-4747-8eb8-78e9f1524a1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:21.316065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.316022 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-podinfo\") pod \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " Nov 03 02:18:21.316065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.316049 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-data" pod "2394b438-fc9a-4747-8eb8-78e9f1524a1d" (UID: "2394b438-fc9a-4747-8eb8-78e9f1524a1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:21.316065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.316061 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2394b438-fc9a-4747-8eb8-78e9f1524a1d-kube-api-access-g2w2k\" (UniqueName: \"kubernetes.io/projected/2394b438-fc9a-4747-8eb8-78e9f1524a1d-kube-api-access-g2w2k\") pod \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " Nov 03 02:18:21.316200 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.316083 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istiod-ca-cert\") pod \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " Nov 03 02:18:21.316200 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.316118 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-credential-socket\") pod \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\" (UID: \"2394b438-fc9a-4747-8eb8-78e9f1524a1d\") " Nov 03 02:18:21.316315 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.316249 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-workload-certs" pod "2394b438-fc9a-4747-8eb8-78e9f1524a1d" (UID: "2394b438-fc9a-4747-8eb8-78e9f1524a1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:21.316541 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.316518 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-credential-socket" pod "2394b438-fc9a-4747-8eb8-78e9f1524a1d" (UID: "2394b438-fc9a-4747-8eb8-78e9f1524a1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:21.316635 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.316561 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istiod-ca-cert" pod "2394b438-fc9a-4747-8eb8-78e9f1524a1d" (UID: "2394b438-fc9a-4747-8eb8-78e9f1524a1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:18:21.317091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.316805 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.317091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.316828 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.317091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.316842 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.317091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.316856 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.317091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.316870 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.318073 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.318054 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-envoy" pod "2394b438-fc9a-4747-8eb8-78e9f1524a1d" (UID: "2394b438-fc9a-4747-8eb8-78e9f1524a1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:21.318367 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.318346 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2394b438-fc9a-4747-8eb8-78e9f1524a1d-kube-api-access-g2w2k" pod "2394b438-fc9a-4747-8eb8-78e9f1524a1d" (UID: "2394b438-fc9a-4747-8eb8-78e9f1524a1d"). InnerVolumeSpecName "kube-api-access-g2w2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:18:21.318588 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.318570 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-podinfo" pod "2394b438-fc9a-4747-8eb8-78e9f1524a1d" (UID: "2394b438-fc9a-4747-8eb8-78e9f1524a1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:18:21.318661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.318592 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-token" pod "2394b438-fc9a-4747-8eb8-78e9f1524a1d" (UID: "2394b438-fc9a-4747-8eb8-78e9f1524a1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:18:21.341574 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.341534 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" podStartSLOduration=2.463672735 podStartE2EDuration="3.341523148s" totalImagesPullingTime="877.850413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:18:21.339136809 +0000 UTC m=+1583.023245963" watchObservedRunningTime="2026-11-03 02:18:21.341523148 +0000 UTC m=+1583.025632303" Nov 03 02:18:21.417551 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.417501 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30a0bcec-82be-4efd-bd03-87dadb66b21b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30a0bcec-82be-4efd-bd03-87dadb66b21b-istiod-ca-cert\") pod \"30a0bcec-82be-4efd-bd03-87dadb66b21b\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " Nov 03 02:18:21.417730 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.417563 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/547c8528-2b07-475d-a191-f0de8a63a4b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/547c8528-2b07-475d-a191-f0de8a63a4b9-istiod-ca-cert\") pod \"547c8528-2b07-475d-a191-f0de8a63a4b9\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " Nov 03 02:18:21.417730 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.417625 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-podinfo\") pod \"30a0bcec-82be-4efd-bd03-87dadb66b21b\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " Nov 03 02:18:21.417730 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.417678 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-data\") pod \"547c8528-2b07-475d-a191-f0de8a63a4b9\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " Nov 03 02:18:21.417730 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.417710 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/547c8528-2b07-475d-a191-f0de8a63a4b9-kube-api-access-shks9\" (UniqueName: \"kubernetes.io/projected/547c8528-2b07-475d-a191-f0de8a63a4b9-kube-api-access-shks9\") pod \"547c8528-2b07-475d-a191-f0de8a63a4b9\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " Nov 03 02:18:21.417961 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.417733 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-podinfo\") pod \"547c8528-2b07-475d-a191-f0de8a63a4b9\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " Nov 03 02:18:21.417961 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.417760 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-workload-certs\") pod \"30a0bcec-82be-4efd-bd03-87dadb66b21b\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " Nov 03 02:18:21.417961 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.417794 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30a0bcec-82be-4efd-bd03-87dadb66b21b-kube-api-access-5z9t9\" (UniqueName: \"kubernetes.io/projected/30a0bcec-82be-4efd-bd03-87dadb66b21b-kube-api-access-5z9t9\") pod \"30a0bcec-82be-4efd-bd03-87dadb66b21b\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " Nov 03 02:18:21.417961 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.417820 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-envoy\") pod \"30a0bcec-82be-4efd-bd03-87dadb66b21b\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " Nov 03 02:18:21.417961 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.417877 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a0bcec-82be-4efd-bd03-87dadb66b21b-istiod-ca-cert" pod "30a0bcec-82be-4efd-bd03-87dadb66b21b" (UID: "30a0bcec-82be-4efd-bd03-87dadb66b21b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:18:21.417961 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.417917 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-envoy\") pod \"547c8528-2b07-475d-a191-f0de8a63a4b9\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " Nov 03 02:18:21.418347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.417984 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-data\") pod \"30a0bcec-82be-4efd-bd03-87dadb66b21b\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " Nov 03 02:18:21.418347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.418056 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-workload-socket\") pod \"547c8528-2b07-475d-a191-f0de8a63a4b9\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " Nov 03 02:18:21.418347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.417997 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/547c8528-2b07-475d-a191-f0de8a63a4b9-istiod-ca-cert" pod "547c8528-2b07-475d-a191-f0de8a63a4b9" (UID: "547c8528-2b07-475d-a191-f0de8a63a4b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:18:21.418347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.418107 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-token\" (UniqueName: \"kubernetes.io/projected/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-token\") pod \"547c8528-2b07-475d-a191-f0de8a63a4b9\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " Nov 03 02:18:21.418347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.418173 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-credential-socket\") pod \"547c8528-2b07-475d-a191-f0de8a63a4b9\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " Nov 03 02:18:21.418347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.418179 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-data" pod "547c8528-2b07-475d-a191-f0de8a63a4b9" (UID: "547c8528-2b07-475d-a191-f0de8a63a4b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:21.418347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.418228 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-token\" (UniqueName: \"kubernetes.io/projected/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-token\") pod \"30a0bcec-82be-4efd-bd03-87dadb66b21b\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " Nov 03 02:18:21.418347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.418274 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-workload-socket\") pod \"30a0bcec-82be-4efd-bd03-87dadb66b21b\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " Nov 03 02:18:21.418347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.418315 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-credential-socket\") pod \"30a0bcec-82be-4efd-bd03-87dadb66b21b\" (UID: \"30a0bcec-82be-4efd-bd03-87dadb66b21b\") " Nov 03 02:18:21.418347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.418345 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-workload-certs\") pod \"547c8528-2b07-475d-a191-f0de8a63a4b9\" (UID: \"547c8528-2b07-475d-a191-f0de8a63a4b9\") " Nov 03 02:18:21.418926 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.418499 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-workload-certs" pod "30a0bcec-82be-4efd-bd03-87dadb66b21b" (UID: "30a0bcec-82be-4efd-bd03-87dadb66b21b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:21.419016 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.418996 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.419078 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.419021 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30a0bcec-82be-4efd-bd03-87dadb66b21b-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.419078 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.419037 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/547c8528-2b07-475d-a191-f0de8a63a4b9-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.419078 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.419052 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.419078 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.419067 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2394b438-fc9a-4747-8eb8-78e9f1524a1d-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.419358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.419081 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2w2k\" (UniqueName: \"kubernetes.io/projected/2394b438-fc9a-4747-8eb8-78e9f1524a1d-kube-api-access-g2w2k\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.419358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.419079 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-data" pod "30a0bcec-82be-4efd-bd03-87dadb66b21b" (UID: "30a0bcec-82be-4efd-bd03-87dadb66b21b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:21.419358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.419095 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.419358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.419112 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.419358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.419319 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-workload-socket" pod "30a0bcec-82be-4efd-bd03-87dadb66b21b" (UID: "30a0bcec-82be-4efd-bd03-87dadb66b21b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:21.419358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.419329 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-workload-certs" pod "547c8528-2b07-475d-a191-f0de8a63a4b9" (UID: "547c8528-2b07-475d-a191-f0de8a63a4b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:21.419740 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.419450 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-credential-socket" pod "30a0bcec-82be-4efd-bd03-87dadb66b21b" (UID: "30a0bcec-82be-4efd-bd03-87dadb66b21b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:21.419740 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.419555 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-workload-socket" pod "547c8528-2b07-475d-a191-f0de8a63a4b9" (UID: "547c8528-2b07-475d-a191-f0de8a63a4b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:21.419740 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.419614 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-credential-socket" pod "547c8528-2b07-475d-a191-f0de8a63a4b9" (UID: "547c8528-2b07-475d-a191-f0de8a63a4b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:21.420592 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.420564 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547c8528-2b07-475d-a191-f0de8a63a4b9-kube-api-access-shks9" pod "547c8528-2b07-475d-a191-f0de8a63a4b9" (UID: "547c8528-2b07-475d-a191-f0de8a63a4b9"). InnerVolumeSpecName "kube-api-access-shks9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:18:21.420758 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.420738 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-podinfo" pod "30a0bcec-82be-4efd-bd03-87dadb66b21b" (UID: "30a0bcec-82be-4efd-bd03-87dadb66b21b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:18:21.420862 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.420837 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-podinfo" pod "547c8528-2b07-475d-a191-f0de8a63a4b9" (UID: "547c8528-2b07-475d-a191-f0de8a63a4b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:18:21.421266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.421240 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-token" pod "30a0bcec-82be-4efd-bd03-87dadb66b21b" (UID: "30a0bcec-82be-4efd-bd03-87dadb66b21b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:18:21.421692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.421671 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-token" pod "547c8528-2b07-475d-a191-f0de8a63a4b9" (UID: "547c8528-2b07-475d-a191-f0de8a63a4b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:18:21.422334 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.422306 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a0bcec-82be-4efd-bd03-87dadb66b21b-kube-api-access-5z9t9" pod "30a0bcec-82be-4efd-bd03-87dadb66b21b" (UID: "30a0bcec-82be-4efd-bd03-87dadb66b21b"). InnerVolumeSpecName "kube-api-access-5z9t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:18:21.422568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.422542 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-envoy" pod "30a0bcec-82be-4efd-bd03-87dadb66b21b" (UID: "30a0bcec-82be-4efd-bd03-87dadb66b21b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:21.423115 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.423090 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-envoy" pod "547c8528-2b07-475d-a191-f0de8a63a4b9" (UID: "547c8528-2b07-475d-a191-f0de8a63a4b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:21.521283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.521243 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.521283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.521283 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shks9\" (UniqueName: \"kubernetes.io/projected/547c8528-2b07-475d-a191-f0de8a63a4b9-kube-api-access-shks9\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.521492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.521300 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.521492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.521314 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z9t9\" (UniqueName: \"kubernetes.io/projected/30a0bcec-82be-4efd-bd03-87dadb66b21b-kube-api-access-5z9t9\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.521492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.521328 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.521492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.521341 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.521492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.521354 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.521492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.521367 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.521492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.521380 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/547c8528-2b07-475d-a191-f0de8a63a4b9-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.521492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.521392 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.521492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.521405 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30a0bcec-82be-4efd-bd03-87dadb66b21b-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.521492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.521418 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.521492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.521442 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a0bcec-82be-4efd-bd03-87dadb66b21b-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:21.521492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:21.521456 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/547c8528-2b07-475d-a191-f0de8a63a4b9-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:22.280366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:22.270800 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" Nov 03 02:18:22.280366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:22.270875 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv" event={"ID":"30a0bcec-82be-4efd-bd03-87dadb66b21b","Type":"ContainerDied","Data":"a9788f5fdfda26bf8a6f3a69ca111e116bad9f9598e34cf46c317498b753f432"} Nov 03 02:18:22.280366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:22.270927 5463 scope.go:122] "RemoveContainer" containerID="4ea19b22cb34eb9d62384865369b7a81ebabed16b32a994dde67b2e4ca656637" Nov 03 02:18:22.284302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:22.283590 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms" Nov 03 02:18:22.294607 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:22.284467 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" Nov 03 02:18:22.294607 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:22.293739 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p" event={"ID":"547c8528-2b07-475d-a191-f0de8a63a4b9","Type":"ContainerDied","Data":"8f5398e2385aa7bcd48a7823b930eef82410af0f7387dfe50363741a9b09ae80"} Nov 03 02:18:22.301735 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:22.301717 5463 scope.go:122] "RemoveContainer" containerID="4ff036340ad598d738328bc2046c567f0dde8508f0aa04e6d148a235e8a3bb2c" Nov 03 02:18:22.445728 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:22.445698 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv"] Nov 03 02:18:22.488861 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:22.488830 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-4vmdv"] Nov 03 02:18:22.597727 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:22.597637 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p"] Nov 03 02:18:22.600629 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:22.600606 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-8cv6p"] Nov 03 02:18:22.710334 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:22.710297 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms"] Nov 03 02:18:22.713294 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:22.713267 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-pj9ms"] Nov 03 02:18:23.816246 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:23.816176 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:24.444309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:24.444279 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2394b438-fc9a-4747-8eb8-78e9f1524a1d" path="/var/lib/kubelet/pods/2394b438-fc9a-4747-8eb8-78e9f1524a1d/volumes" Nov 03 02:18:24.444805 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:24.444791 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30a0bcec-82be-4efd-bd03-87dadb66b21b" path="/var/lib/kubelet/pods/30a0bcec-82be-4efd-bd03-87dadb66b21b/volumes" Nov 03 02:18:24.445292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:24.445279 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="547c8528-2b07-475d-a191-f0de8a63a4b9" path="/var/lib/kubelet/pods/547c8528-2b07-475d-a191-f0de8a63a4b9/volumes" Nov 03 02:18:27.301513 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:27.301482 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" Nov 03 02:18:27.959115 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:27.959083 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:29.711181 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:29.711150 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:34.068160 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.068089 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:34.429511 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.429482 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:34.548622 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.548594 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f"] Nov 03 02:18:34.549263 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.549245 5463 cpu_manager.go:414] "containerMap: removing container" podUID="30a0bcec-82be-4efd-bd03-87dadb66b21b" containerName="istio-proxy" Nov 03 02:18:34.549323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.549266 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30a0bcec-82be-4efd-bd03-87dadb66b21b" containerName="istio-proxy" Nov 03 02:18:34.549323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.549289 5463 cpu_manager.go:414] "containerMap: removing container" podUID="547c8528-2b07-475d-a191-f0de8a63a4b9" containerName="istio-proxy" Nov 03 02:18:34.549323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.549294 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="547c8528-2b07-475d-a191-f0de8a63a4b9" containerName="istio-proxy" Nov 03 02:18:34.549323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.549307 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2394b438-fc9a-4747-8eb8-78e9f1524a1d" containerName="istio-proxy" Nov 03 02:18:34.549323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.549313 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2394b438-fc9a-4747-8eb8-78e9f1524a1d" containerName="istio-proxy" Nov 03 02:18:34.549499 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.549463 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="30a0bcec-82be-4efd-bd03-87dadb66b21b" containerName="istio-proxy" Nov 03 02:18:34.549499 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.549475 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="547c8528-2b07-475d-a191-f0de8a63a4b9" containerName="istio-proxy" Nov 03 02:18:34.549499 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.549482 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2394b438-fc9a-4747-8eb8-78e9f1524a1d" containerName="istio-proxy" Nov 03 02:18:34.554981 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.554957 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.614336 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.614300 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-token\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.614585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.614562 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whbm5\" (UniqueName: \"kubernetes.io/projected/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-kube-api-access-whbm5\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.614745 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.614716 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istiod-ca-cert\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.614875 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.614814 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-credential-socket\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.614945 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.614904 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-workload-socket\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.615011 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.614945 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-workload-certs\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.615070 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.615013 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-data\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.615184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.615155 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-podinfo\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.615332 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.615246 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-envoy\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.623466 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.623433 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f66575-openshift-default-dockercfg-cc2xd\"" Nov 03 02:18:34.680236 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.670850 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f"] Nov 03 02:18:34.776724 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.776693 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-credential-socket\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.777040 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.777018 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-workload-socket\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.781184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.777293 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-workload-certs\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.781444 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.781423 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-data\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.781955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.781937 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-podinfo\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.782551 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.782521 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-envoy\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.783274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.783254 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-token\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.783426 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.783407 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whbm5\" (UniqueName: \"kubernetes.io/projected/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-kube-api-access-whbm5\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.783571 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.783548 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:34.783654 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.777648 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-workload-certs\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.783720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.783552 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istiod-ca-cert\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.783805 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.781871 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-data\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.783870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.777391 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-workload-socket\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.783949 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.777116 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-credential-socket\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.784801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.784758 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istiod-ca-cert\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.799207 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.797895 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-podinfo\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.799809 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.799593 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-envoy\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.843682 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.843655 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whbm5\" (UniqueName: \"kubernetes.io/projected/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-kube-api-access-whbm5\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.843932 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.843911 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-token\") pod \"e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.870481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.870449 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:34.935781 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:34.935589 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:35.293568 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:18:35.293532 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80c313a_aa7c_4fd4_8c2b_7c5d2ecc7dbd.slice/crio-c4b4888c899512e766cbf15df08dde6f976ee659de5da11d24f57c467a3f30da WatchSource:0}: Error finding container c4b4888c899512e766cbf15df08dde6f976ee659de5da11d24f57c467a3f30da: Status 404 returned error can't find the container with id c4b4888c899512e766cbf15df08dde6f976ee659de5da11d24f57c467a3f30da Nov 03 02:18:35.296571 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:35.296537 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:18:35.296682 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:35.296602 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:18:35.296682 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:35.296632 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:18:35.297280 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:35.297253 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f"] Nov 03 02:18:35.339063 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:35.339038 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:35.410223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:35.409166 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" event={"ID":"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd","Type":"ContainerStarted","Data":"c4b4888c899512e766cbf15df08dde6f976ee659de5da11d24f57c467a3f30da"} Nov 03 02:18:35.496238 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:35.496078 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:36.418539 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:36.418505 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" event={"ID":"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd","Type":"ContainerStarted","Data":"99dca0b9f302869d45fae0b9555ae0f00430ab14f5c2a3ff1b6d01822a39155a"} Nov 03 02:18:36.536428 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:36.536381 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" podStartSLOduration=2.536367394 podStartE2EDuration="2.536367394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:18:36.534963088 +0000 UTC m=+1598.219072244" watchObservedRunningTime="2026-11-03 02:18:36.536367394 +0000 UTC m=+1598.220476548" Nov 03 02:18:36.871619 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:36.871522 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:36.878044 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:36.878018 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:36.911075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:36.911049 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:37.402399 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:37.402365 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:37.425080 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:37.425054 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:37.426038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:37.426017 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:18:38.747717 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:38.747678 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:39.272629 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.272597 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls"] Nov 03 02:18:39.277357 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.277334 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" Nov 03 02:18:39.329299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.329249 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84"] Nov 03 02:18:39.343928 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.341621 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls"] Nov 03 02:18:39.343928 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.341748 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.369072 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.369033 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d549eff-db94-49f8-bff2-4e7f7d24071d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls\" (UID: \"0d549eff-db94-49f8-bff2-4e7f7d24071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" Nov 03 02:18:39.371066 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.371044 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plthn\" (UniqueName: \"kubernetes.io/projected/0d549eff-db94-49f8-bff2-4e7f7d24071d-kube-api-access-plthn\") pod \"maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls\" (UID: \"0d549eff-db94-49f8-bff2-4e7f7d24071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" Nov 03 02:18:39.371355 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.371336 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d549eff-db94-49f8-bff2-4e7f7d24071d-metrics-certs\") pod \"maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls\" (UID: \"0d549eff-db94-49f8-bff2-4e7f7d24071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" Nov 03 02:18:39.371626 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.371608 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d549eff-db94-49f8-bff2-4e7f7d24071d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls\" (UID: \"0d549eff-db94-49f8-bff2-4e7f7d24071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" Nov 03 02:18:39.382215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.380826 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ff26f5-serving-cert\"" Nov 03 02:18:39.405217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.402880 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b3bbbc-openshift-default-dockercfg-lx5rf\"" Nov 03 02:18:39.453974 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.452316 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84"] Nov 03 02:18:39.455540 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.455069 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:39.476227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.472769 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-podinfo\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.476227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.472829 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d549eff-db94-49f8-bff2-4e7f7d24071d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls\" (UID: \"0d549eff-db94-49f8-bff2-4e7f7d24071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" Nov 03 02:18:39.476227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.472887 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plthn\" (UniqueName: \"kubernetes.io/projected/0d549eff-db94-49f8-bff2-4e7f7d24071d-kube-api-access-plthn\") pod \"maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls\" (UID: \"0d549eff-db94-49f8-bff2-4e7f7d24071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" Nov 03 02:18:39.476227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.472928 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istiod-ca-cert\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.476227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.472968 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzkpc\" (UniqueName: \"kubernetes.io/projected/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-kube-api-access-kzkpc\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.476227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.473019 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-token\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.476227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.473049 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d549eff-db94-49f8-bff2-4e7f7d24071d-metrics-certs\") pod \"maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls\" (UID: \"0d549eff-db94-49f8-bff2-4e7f7d24071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" Nov 03 02:18:39.476227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.473093 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-credential-socket\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.476227 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:18:39.473721 5463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-serving-cert: secret "maas-api-e2e-ipp-a-ff26f5-serving-cert" not found Nov 03 02:18:39.476227 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:18:39.475038 5463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d549eff-db94-49f8-bff2-4e7f7d24071d-maas-api-tls podName:0d549eff-db94-49f8-bff2-4e7f7d24071d nodeName:}" failed. No retries permitted until 2026-11-03 02:18:39.975012759 +0000 UTC m=+1601.659121904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0d549eff-db94-49f8-bff2-4e7f7d24071d-maas-api-tls") pod "maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" (UID: "0d549eff-db94-49f8-bff2-4e7f7d24071d") : secret "maas-api-e2e-ipp-a-ff26f5-serving-cert" not found Nov 03 02:18:39.476227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.475118 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-workload-socket\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.476227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.475146 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-workload-certs\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.476227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.475170 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-envoy\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.476227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.475278 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d549eff-db94-49f8-bff2-4e7f7d24071d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls\" (UID: \"0d549eff-db94-49f8-bff2-4e7f7d24071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" Nov 03 02:18:39.476227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.475351 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-data\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.476227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.476103 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d549eff-db94-49f8-bff2-4e7f7d24071d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls\" (UID: \"0d549eff-db94-49f8-bff2-4e7f7d24071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" Nov 03 02:18:39.487426 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.482533 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d549eff-db94-49f8-bff2-4e7f7d24071d-metrics-certs\") pod \"maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls\" (UID: \"0d549eff-db94-49f8-bff2-4e7f7d24071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" Nov 03 02:18:39.533059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.532979 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plthn\" (UniqueName: \"kubernetes.io/projected/0d549eff-db94-49f8-bff2-4e7f7d24071d-kube-api-access-plthn\") pod \"maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls\" (UID: \"0d549eff-db94-49f8-bff2-4e7f7d24071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" Nov 03 02:18:39.577681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.576536 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-data\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.577681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.576594 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-podinfo\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.577681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.576674 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istiod-ca-cert\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.577681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.576709 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzkpc\" (UniqueName: \"kubernetes.io/projected/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-kube-api-access-kzkpc\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.577681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.576769 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-token\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.577681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.576836 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-credential-socket\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.577681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.576899 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-data\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.577681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.576922 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-workload-socket\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.577681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.576947 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-workload-certs\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.577681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.576969 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-envoy\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.577681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.577473 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istiod-ca-cert\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.577681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.577605 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-credential-socket\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.578292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.577743 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-workload-socket\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.578292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.577896 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-workload-certs\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.579834 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.579785 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-envoy\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.580097 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.580053 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-podinfo\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.611451 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.611412 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:39.635654 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.635624 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-token\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.637154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.637126 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzkpc\" (UniqueName: \"kubernetes.io/projected/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-kube-api-access-kzkpc\") pod \"e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.688728 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.688688 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:39.769694 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.768548 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:39.914929 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.914887 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:39.992391 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.992358 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d549eff-db94-49f8-bff2-4e7f7d24071d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls\" (UID: \"0d549eff-db94-49f8-bff2-4e7f7d24071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" Nov 03 02:18:39.995041 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:39.995008 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d549eff-db94-49f8-bff2-4e7f7d24071d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls\" (UID: \"0d549eff-db94-49f8-bff2-4e7f7d24071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" Nov 03 02:18:40.066373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:40.066299 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:40.132068 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:18:40.132039 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62ad394_77bf_4a2d_8ec1_ed0f6238b3de.slice/crio-4e62ecc861209fa7aa927f06751066f49b99e705c7836e14b66662f8b5935254 WatchSource:0}: Error finding container 4e62ecc861209fa7aa927f06751066f49b99e705c7836e14b66662f8b5935254: Status 404 returned error can't find the container with id 4e62ecc861209fa7aa927f06751066f49b99e705c7836e14b66662f8b5935254 Nov 03 02:18:40.134060 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:40.134030 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84"] Nov 03 02:18:40.134657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:40.134621 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:18:40.134754 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:40.134683 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:18:40.134754 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:40.134714 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:18:40.193659 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:40.193637 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" Nov 03 02:18:40.234624 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:40.234581 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:40.485585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:40.485552 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" event={"ID":"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de","Type":"ContainerStarted","Data":"7b9e02876cb8942f0f9fc54b2eccdbf928cae3fd83221b37f283bb3a79523746"} Nov 03 02:18:40.485794 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:40.485602 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" event={"ID":"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de","Type":"ContainerStarted","Data":"4e62ecc861209fa7aa927f06751066f49b99e705c7836e14b66662f8b5935254"} Nov 03 02:18:40.584370 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:18:40.584326 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d549eff_db94_49f8_bff2_4e7f7d24071d.slice/crio-e60dec01c86a2faa9134492e91d248417c1d26ba2e17295b8d307479a07b48c2 WatchSource:0}: Error finding container e60dec01c86a2faa9134492e91d248417c1d26ba2e17295b8d307479a07b48c2: Status 404 returned error can't find the container with id e60dec01c86a2faa9134492e91d248417c1d26ba2e17295b8d307479a07b48c2 Nov 03 02:18:40.589942 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:40.589917 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls"] Nov 03 02:18:40.603478 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:40.603428 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" podStartSLOduration=1.603410599 podStartE2EDuration="1.603410599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:18:40.602372607 +0000 UTC m=+1602.286481764" watchObservedRunningTime="2026-11-03 02:18:40.603410599 +0000 UTC m=+1602.287519754" Nov 03 02:18:40.689490 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:40.689450 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:41.494997 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:41.494953 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" event={"ID":"0d549eff-db94-49f8-bff2-4e7f7d24071d","Type":"ContainerStarted","Data":"e60dec01c86a2faa9134492e91d248417c1d26ba2e17295b8d307479a07b48c2"} Nov 03 02:18:41.695526 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:41.695498 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:42.503917 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:42.503879 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" event={"ID":"0d549eff-db94-49f8-bff2-4e7f7d24071d","Type":"ContainerStarted","Data":"43129c8669d6dbe19e05bf587dbc75470d6e9c93874509ea5ce8209f24bf131a"} Nov 03 02:18:42.504442 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:42.504080 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:42.504442 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:42.504118 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" Nov 03 02:18:42.505078 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:42.505060 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:18:42.752458 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:42.752279 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" podStartSLOduration=2.833755118 podStartE2EDuration="3.752259211s" totalImagesPullingTime="918.504093ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:18:42.632233667 +0000 UTC m=+1604.316342820" watchObservedRunningTime="2026-11-03 02:18:42.752259211 +0000 UTC m=+1604.436368370" Nov 03 02:18:43.019315 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:18:43.019281 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:18:43.279624 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:43.279529 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:43.459741 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:43.459705 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:43.687484 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:43.687443 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg"] Nov 03 02:18:43.692046 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:43.692023 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg" Nov 03 02:18:43.744410 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:43.744382 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg"] Nov 03 02:18:43.746032 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:43.746009 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5089d5c-3c50-40d7-a89e-90946d0404e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg\" (UID: \"c5089d5c-3c50-40d7-a89e-90946d0404e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg" Nov 03 02:18:43.746237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:43.746218 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltxjs\" (UniqueName: \"kubernetes.io/projected/c5089d5c-3c50-40d7-a89e-90946d0404e3-kube-api-access-ltxjs\") pod \"payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg\" (UID: \"c5089d5c-3c50-40d7-a89e-90946d0404e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg" Nov 03 02:18:43.812244 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:43.801545 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f66575\"" Nov 03 02:18:43.827231 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:43.824638 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f66575-dockercfg-rg2ls\"" Nov 03 02:18:43.855215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:43.854908 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltxjs\" (UniqueName: \"kubernetes.io/projected/c5089d5c-3c50-40d7-a89e-90946d0404e3-kube-api-access-ltxjs\") pod \"payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg\" (UID: \"c5089d5c-3c50-40d7-a89e-90946d0404e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg" Nov 03 02:18:43.855215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:43.855105 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5089d5c-3c50-40d7-a89e-90946d0404e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg\" (UID: \"c5089d5c-3c50-40d7-a89e-90946d0404e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg" Nov 03 02:18:43.865228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:43.856216 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5089d5c-3c50-40d7-a89e-90946d0404e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg\" (UID: \"c5089d5c-3c50-40d7-a89e-90946d0404e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg" Nov 03 02:18:44.011314 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:44.011213 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltxjs\" (UniqueName: \"kubernetes.io/projected/c5089d5c-3c50-40d7-a89e-90946d0404e3-kube-api-access-ltxjs\") pod \"payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg\" (UID: \"c5089d5c-3c50-40d7-a89e-90946d0404e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg" Nov 03 02:18:44.031886 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:44.031842 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q"] Nov 03 02:18:44.041304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:44.041275 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q"] Nov 03 02:18:44.041459 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:44.041368 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q" Nov 03 02:18:44.070520 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:44.070484 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v6nc\" (UniqueName: \"kubernetes.io/projected/b9f81dca-b2c4-4d87-afc1-08db93b24e6b-kube-api-access-8v6nc\") pod \"payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q\" (UID: \"b9f81dca-b2c4-4d87-afc1-08db93b24e6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q" Nov 03 02:18:44.089429 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:44.089398 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f81dca-b2c4-4d87-afc1-08db93b24e6b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q\" (UID: \"b9f81dca-b2c4-4d87-afc1-08db93b24e6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q" Nov 03 02:18:44.191345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:44.191315 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v6nc\" (UniqueName: \"kubernetes.io/projected/b9f81dca-b2c4-4d87-afc1-08db93b24e6b-kube-api-access-8v6nc\") pod \"payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q\" (UID: \"b9f81dca-b2c4-4d87-afc1-08db93b24e6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q" Nov 03 02:18:44.191530 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:44.191448 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f81dca-b2c4-4d87-afc1-08db93b24e6b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q\" (UID: \"b9f81dca-b2c4-4d87-afc1-08db93b24e6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q" Nov 03 02:18:44.192364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:44.192333 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f81dca-b2c4-4d87-afc1-08db93b24e6b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q\" (UID: \"b9f81dca-b2c4-4d87-afc1-08db93b24e6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q" Nov 03 02:18:44.281005 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:44.280925 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v6nc\" (UniqueName: \"kubernetes.io/projected/b9f81dca-b2c4-4d87-afc1-08db93b24e6b-kube-api-access-8v6nc\") pod \"payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q\" (UID: \"b9f81dca-b2c4-4d87-afc1-08db93b24e6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q" Nov 03 02:18:44.308679 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:44.308654 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg" Nov 03 02:18:44.401950 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:44.401888 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q" Nov 03 02:18:44.719851 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:18:44.719806 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5089d5c_3c50_40d7_a89e_90946d0404e3.slice/crio-2e65de484ec09a6896a540ab602c2cc695d53dce1b04332b19aa754390a92582 WatchSource:0}: Error finding container 2e65de484ec09a6896a540ab602c2cc695d53dce1b04332b19aa754390a92582: Status 404 returned error can't find the container with id 2e65de484ec09a6896a540ab602c2cc695d53dce1b04332b19aa754390a92582 Nov 03 02:18:44.723249 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:44.723226 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg"] Nov 03 02:18:44.816008 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:18:44.815978 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f81dca_b2c4_4d87_afc1_08db93b24e6b.slice/crio-4481dcdbe2d65f9e15b542604b9cf28191968ec8bf7d962de22153ccdb49d0dc WatchSource:0}: Error finding container 4481dcdbe2d65f9e15b542604b9cf28191968ec8bf7d962de22153ccdb49d0dc: Status 404 returned error can't find the container with id 4481dcdbe2d65f9e15b542604b9cf28191968ec8bf7d962de22153ccdb49d0dc Nov 03 02:18:44.820159 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:44.820140 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q"] Nov 03 02:18:44.886221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:44.883911 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:45.267214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:45.266669 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:45.481753 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:45.481722 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:45.529273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:45.529154 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg" event={"ID":"c5089d5c-3c50-40d7-a89e-90946d0404e3","Type":"ContainerStarted","Data":"7a0090a7e3b5f5108a33324e9dd4a1ad6533800dd51796edb73ce9d175adc142"} Nov 03 02:18:45.529273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:45.529231 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg" event={"ID":"c5089d5c-3c50-40d7-a89e-90946d0404e3","Type":"ContainerStarted","Data":"2e65de484ec09a6896a540ab602c2cc695d53dce1b04332b19aa754390a92582"} Nov 03 02:18:45.529273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:45.529260 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg" Nov 03 02:18:45.530887 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:45.530857 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q" event={"ID":"b9f81dca-b2c4-4d87-afc1-08db93b24e6b","Type":"ContainerStarted","Data":"0a80d49f8de0ea770c2e3aafdb89914e102f1ab4cdb9e41923d6f051474a70f0"} Nov 03 02:18:45.531029 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:45.530907 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q" event={"ID":"b9f81dca-b2c4-4d87-afc1-08db93b24e6b","Type":"ContainerStarted","Data":"4481dcdbe2d65f9e15b542604b9cf28191968ec8bf7d962de22153ccdb49d0dc"} Nov 03 02:18:45.531029 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:45.530937 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q" Nov 03 02:18:45.647118 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:45.647065 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg" podStartSLOduration=2.6470503560000003 podStartE2EDuration="2.647050356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:18:45.644177428 +0000 UTC m=+1607.328286584" watchObservedRunningTime="2026-11-03 02:18:45.647050356 +0000 UTC m=+1607.331159511" Nov 03 02:18:45.673218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:45.670712 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:45.772055 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:45.769732 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q" podStartSLOduration=2.769710114 podStartE2EDuration="2.769710114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:18:45.761976399 +0000 UTC m=+1607.446085554" watchObservedRunningTime="2026-11-03 02:18:45.769710114 +0000 UTC m=+1607.453819271" Nov 03 02:18:48.517478 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:48.517450 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" Nov 03 02:18:49.917697 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:49.917660 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:50.572280 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:50.572248 5463 generic.go:356] "Generic (PLEG): container finished" podID="84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" containerID="e1667115830b7fe3fae41cf011b4b63a9805ba45cb461ee9652d11320dbda32c" exitCode=0 Nov 03 02:18:50.572451 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:50.572296 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" event={"ID":"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e","Type":"ContainerDied","Data":"e1667115830b7fe3fae41cf011b4b63a9805ba45cb461ee9652d11320dbda32c"} Nov 03 02:18:50.572694 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:50.572681 5463 scope.go:122] "RemoveContainer" containerID="e1667115830b7fe3fae41cf011b4b63a9805ba45cb461ee9652d11320dbda32c" Nov 03 02:18:50.574289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:50.574257 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:18:50.574391 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:50.574326 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:18:50.574391 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:50.574360 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:18:50.588474 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:50.588439 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:50.768614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:50.768572 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:50.972346 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:50.972316 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt"] Nov 03 02:18:50.975897 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:50.975878 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt" Nov 03 02:18:51.029380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.029344 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b3bbbc\"" Nov 03 02:18:51.029604 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.029579 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt"] Nov 03 02:18:51.101656 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.101618 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td9l8\" (UniqueName: \"kubernetes.io/projected/cd50e030-93e4-41ab-afc4-21f098a66911-kube-api-access-td9l8\") pod \"payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt\" (UID: \"cd50e030-93e4-41ab-afc4-21f098a66911\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt" Nov 03 02:18:51.101957 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.101931 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd50e030-93e4-41ab-afc4-21f098a66911-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt\" (UID: \"cd50e030-93e4-41ab-afc4-21f098a66911\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt" Nov 03 02:18:51.215614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.213341 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td9l8\" (UniqueName: \"kubernetes.io/projected/cd50e030-93e4-41ab-afc4-21f098a66911-kube-api-access-td9l8\") pod \"payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt\" (UID: \"cd50e030-93e4-41ab-afc4-21f098a66911\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt" Nov 03 02:18:51.215614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.213500 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd50e030-93e4-41ab-afc4-21f098a66911-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt\" (UID: \"cd50e030-93e4-41ab-afc4-21f098a66911\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt" Nov 03 02:18:51.215614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.215079 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd50e030-93e4-41ab-afc4-21f098a66911-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt\" (UID: \"cd50e030-93e4-41ab-afc4-21f098a66911\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt" Nov 03 02:18:51.217969 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.217926 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g"] Nov 03 02:18:51.230378 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.230295 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g"] Nov 03 02:18:51.231162 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.231123 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g" Nov 03 02:18:51.272258 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.272229 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td9l8\" (UniqueName: \"kubernetes.io/projected/cd50e030-93e4-41ab-afc4-21f098a66911-kube-api-access-td9l8\") pod \"payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt\" (UID: \"cd50e030-93e4-41ab-afc4-21f098a66911\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt" Nov 03 02:18:51.290935 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.290464 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b3bbbc-dockercfg-qmx4l\"" Nov 03 02:18:51.294844 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.293485 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt" Nov 03 02:18:51.315168 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.315132 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzlmm\" (UniqueName: \"kubernetes.io/projected/fa734878-76ae-43cd-b47b-d5752ded7a8b-kube-api-access-mzlmm\") pod \"payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g\" (UID: \"fa734878-76ae-43cd-b47b-d5752ded7a8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g" Nov 03 02:18:51.315614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.315591 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa734878-76ae-43cd-b47b-d5752ded7a8b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g\" (UID: \"fa734878-76ae-43cd-b47b-d5752ded7a8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g" Nov 03 02:18:51.419259 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.417438 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzlmm\" (UniqueName: \"kubernetes.io/projected/fa734878-76ae-43cd-b47b-d5752ded7a8b-kube-api-access-mzlmm\") pod \"payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g\" (UID: \"fa734878-76ae-43cd-b47b-d5752ded7a8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g" Nov 03 02:18:51.419259 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.417518 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa734878-76ae-43cd-b47b-d5752ded7a8b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g\" (UID: \"fa734878-76ae-43cd-b47b-d5752ded7a8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g" Nov 03 02:18:51.419259 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.418827 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa734878-76ae-43cd-b47b-d5752ded7a8b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g\" (UID: \"fa734878-76ae-43cd-b47b-d5752ded7a8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g" Nov 03 02:18:51.476848 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.476799 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzlmm\" (UniqueName: \"kubernetes.io/projected/fa734878-76ae-43cd-b47b-d5752ded7a8b-kube-api-access-mzlmm\") pod \"payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g\" (UID: \"fa734878-76ae-43cd-b47b-d5752ded7a8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g" Nov 03 02:18:51.572265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.569445 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g" Nov 03 02:18:51.585930 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.585890 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" event={"ID":"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e","Type":"ContainerStarted","Data":"1893449848da4a7f7aa9a010f348b47cff64039d2c6a76535194214c3e0ed51b"} Nov 03 02:18:51.714542 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:18:51.714508 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd50e030_93e4_41ab_afc4_21f098a66911.slice/crio-e374370690594d8cf21bfa85589b512df6bdd9d6d595026f2d6234f707242aaf WatchSource:0}: Error finding container e374370690594d8cf21bfa85589b512df6bdd9d6d595026f2d6234f707242aaf: Status 404 returned error can't find the container with id e374370690594d8cf21bfa85589b512df6bdd9d6d595026f2d6234f707242aaf Nov 03 02:18:51.717879 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.717850 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt"] Nov 03 02:18:51.905953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.905922 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:18:51.906157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.906137 5463 patch_prober.go:29] interesting pod/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 03 02:18:51.906253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.906207 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" podUID="84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 03 02:18:51.944103 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:18:51.944077 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa734878_76ae_43cd_b47b_d5752ded7a8b.slice/crio-8d6427c396d0efe37a647935f99ce5e5ce43fee3528ac0014010fcd9173ba11c WatchSource:0}: Error finding container 8d6427c396d0efe37a647935f99ce5e5ce43fee3528ac0014010fcd9173ba11c: Status 404 returned error can't find the container with id 8d6427c396d0efe37a647935f99ce5e5ce43fee3528ac0014010fcd9173ba11c Nov 03 02:18:51.956247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:51.947569 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g"] Nov 03 02:18:52.607759 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:52.607718 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g" event={"ID":"fa734878-76ae-43cd-b47b-d5752ded7a8b","Type":"ContainerStarted","Data":"f7b49f94752a2b0a31f2761f4d519ec8632d5b33ec97326636586abd197b5093"} Nov 03 02:18:52.608237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:52.607802 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g" event={"ID":"fa734878-76ae-43cd-b47b-d5752ded7a8b","Type":"ContainerStarted","Data":"8d6427c396d0efe37a647935f99ce5e5ce43fee3528ac0014010fcd9173ba11c"} Nov 03 02:18:52.608237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:52.607827 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g" Nov 03 02:18:52.609884 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:52.609859 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt" event={"ID":"cd50e030-93e4-41ab-afc4-21f098a66911","Type":"ContainerStarted","Data":"725601e93cabfdb737864ad0920c1feee88553145440ede7e4b8168a6c45c2ac"} Nov 03 02:18:52.610064 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:52.610050 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt" event={"ID":"cd50e030-93e4-41ab-afc4-21f098a66911","Type":"ContainerStarted","Data":"e374370690594d8cf21bfa85589b512df6bdd9d6d595026f2d6234f707242aaf"} Nov 03 02:18:52.610146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:52.610137 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt" Nov 03 02:18:52.728782 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:52.728732 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g" podStartSLOduration=1.728719024 podStartE2EDuration="1.728719024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:18:52.725758534 +0000 UTC m=+1614.409867689" watchObservedRunningTime="2026-11-03 02:18:52.728719024 +0000 UTC m=+1614.412828178" Nov 03 02:18:52.844175 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:52.844124 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt" podStartSLOduration=2.844108748 podStartE2EDuration="2.844108748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:18:52.839704647 +0000 UTC m=+1614.523813813" watchObservedRunningTime="2026-11-03 02:18:52.844108748 +0000 UTC m=+1614.528217902" Nov 03 02:18:52.906030 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:52.905999 5463 patch_prober.go:29] interesting pod/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 03 02:18:52.906220 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:52.906055 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" podUID="84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 03 02:18:53.735038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:53.735009 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7241-781b67"] Nov 03 02:18:53.739005 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:53.738987 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7241-781b67" Nov 03 02:18:53.789991 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:53.789950 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7241-781b67"] Nov 03 02:18:53.836273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:53.836226 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-579jc\"" Nov 03 02:18:53.874215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:53.874096 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9bcs\" (UniqueName: \"kubernetes.io/projected/7ead49c5-780b-498a-84ab-bbbadea4d8b7-kube-api-access-l9bcs\") pod \"test-curl-7241-781b67\" (UID: \"7ead49c5-780b-498a-84ab-bbbadea4d8b7\") " pod="opendatahub/test-curl-7241-781b67" Nov 03 02:18:53.905894 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:53.905861 5463 patch_prober.go:29] interesting pod/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 03 02:18:53.906052 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:53.905920 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" podUID="84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 03 02:18:53.978874 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:53.978842 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9bcs\" (UniqueName: \"kubernetes.io/projected/7ead49c5-780b-498a-84ab-bbbadea4d8b7-kube-api-access-l9bcs\") pod \"test-curl-7241-781b67\" (UID: \"7ead49c5-780b-498a-84ab-bbbadea4d8b7\") " pod="opendatahub/test-curl-7241-781b67" Nov 03 02:18:54.037881 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:54.037807 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9bcs\" (UniqueName: \"kubernetes.io/projected/7ead49c5-780b-498a-84ab-bbbadea4d8b7-kube-api-access-l9bcs\") pod \"test-curl-7241-781b67\" (UID: \"7ead49c5-780b-498a-84ab-bbbadea4d8b7\") " pod="opendatahub/test-curl-7241-781b67" Nov 03 02:18:54.077621 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:54.077587 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7241-781b67" Nov 03 02:18:54.476553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:54.476520 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7241-781b67"] Nov 03 02:18:54.626270 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:54.626229 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7241-781b67" event={"ID":"7ead49c5-780b-498a-84ab-bbbadea4d8b7","Type":"ContainerStarted","Data":"b5e815f2818086821e4b2d2e7c6675eef41b5ece6b02f93b4257017c8dc3c32b"} Nov 03 02:18:54.905908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:54.905875 5463 patch_prober.go:29] interesting pod/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 03 02:18:54.906325 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:54.905940 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" podUID="84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 03 02:18:55.015807 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:55.015768 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:55.907886 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:55.907816 5463 patch_prober.go:29] interesting pod/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 03 02:18:55.909156 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:55.909035 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" podUID="84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 03 02:18:56.544181 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:56.544148 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg" Nov 03 02:18:56.544369 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:56.544354 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q" Nov 03 02:18:56.906277 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:56.906245 5463 patch_prober.go:29] interesting pod/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 03 02:18:56.906467 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:56.906304 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" podUID="84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 03 02:18:57.649872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:57.649836 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7241-781b67" event={"ID":"7ead49c5-780b-498a-84ab-bbbadea4d8b7","Type":"ContainerStarted","Data":"3e3d1388550df5448045aeda1efef9bba8c6df4dcf53f4ae6ea9d6638d730b5d"} Nov 03 02:18:57.765691 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:57.765635 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7241-781b67" podStartSLOduration=1.8293132220000001 podStartE2EDuration="4.765617328s" totalImagesPullingTime="2.936304106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:18:57.762056132 +0000 UTC m=+1619.446165287" watchObservedRunningTime="2026-11-03 02:18:57.765617328 +0000 UTC m=+1619.449726485" Nov 03 02:18:57.910918 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:57.910838 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:18:57.911260 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:57.911223 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:18:57.911910 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:57.911893 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:18:59.204454 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:59.204059 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7241-781b67"] Nov 03 02:18:59.208387 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:59.208358 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7241-781b67"] Nov 03 02:18:59.665828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:59.665790 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7241-781b67" podUID="7ead49c5-780b-498a-84ab-bbbadea4d8b7" containerName="test-curl-7241-781b67" containerID="cri-o://3e3d1388550df5448045aeda1efef9bba8c6df4dcf53f4ae6ea9d6638d730b5d" gracePeriod=2 Nov 03 02:18:59.684794 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:59.684756 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:59.717551 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:18:59.717510 5463 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7241-781b67\" is forbidden: User \"system:node:ip-10-0-137-69.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-69.us-west-2.compute.internal' and this object" podUID="7ead49c5-780b-498a-84ab-bbbadea4d8b7" pod="opendatahub/test-curl-7241-781b67" Nov 03 02:18:59.837347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:18:59.837316 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:01.937760 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:01.937707 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7241-781b67" Nov 03 02:19:01.994418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:01.994389 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ead49c5-780b-498a-84ab-bbbadea4d8b7-kube-api-access-l9bcs\" (UniqueName: \"kubernetes.io/projected/7ead49c5-780b-498a-84ab-bbbadea4d8b7-kube-api-access-l9bcs\") pod \"7ead49c5-780b-498a-84ab-bbbadea4d8b7\" (UID: \"7ead49c5-780b-498a-84ab-bbbadea4d8b7\") " Nov 03 02:19:01.996520 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:01.996494 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ead49c5-780b-498a-84ab-bbbadea4d8b7-kube-api-access-l9bcs" pod "7ead49c5-780b-498a-84ab-bbbadea4d8b7" (UID: "7ead49c5-780b-498a-84ab-bbbadea4d8b7"). InnerVolumeSpecName "kube-api-access-l9bcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:19:02.096405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:02.096367 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9bcs\" (UniqueName: \"kubernetes.io/projected/7ead49c5-780b-498a-84ab-bbbadea4d8b7-kube-api-access-l9bcs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:02.443780 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:02.443744 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ead49c5-780b-498a-84ab-bbbadea4d8b7" path="/var/lib/kubelet/pods/7ead49c5-780b-498a-84ab-bbbadea4d8b7/volumes" Nov 03 02:19:02.695724 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:02.695627 5463 generic.go:356] "Generic (PLEG): container finished" podID="7ead49c5-780b-498a-84ab-bbbadea4d8b7" containerID="3e3d1388550df5448045aeda1efef9bba8c6df4dcf53f4ae6ea9d6638d730b5d" exitCode=137 Nov 03 02:19:02.695724 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:02.695667 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7241-781b67" Nov 03 02:19:02.695957 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:02.695732 5463 scope.go:122] "RemoveContainer" containerID="3e3d1388550df5448045aeda1efef9bba8c6df4dcf53f4ae6ea9d6638d730b5d" Nov 03 02:19:02.706296 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:02.706269 5463 scope.go:122] "RemoveContainer" containerID="3e3d1388550df5448045aeda1efef9bba8c6df4dcf53f4ae6ea9d6638d730b5d" Nov 03 02:19:02.706576 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:19:02.706554 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3d1388550df5448045aeda1efef9bba8c6df4dcf53f4ae6ea9d6638d730b5d\": container with ID starting with 3e3d1388550df5448045aeda1efef9bba8c6df4dcf53f4ae6ea9d6638d730b5d not found: ID does not exist" containerID="3e3d1388550df5448045aeda1efef9bba8c6df4dcf53f4ae6ea9d6638d730b5d" Nov 03 02:19:02.706665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:02.706583 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3d1388550df5448045aeda1efef9bba8c6df4dcf53f4ae6ea9d6638d730b5d"} err="failed to get container status \"3e3d1388550df5448045aeda1efef9bba8c6df4dcf53f4ae6ea9d6638d730b5d\": rpc error: code = NotFound desc = could not find container \"3e3d1388550df5448045aeda1efef9bba8c6df4dcf53f4ae6ea9d6638d730b5d\": container with ID starting with 3e3d1388550df5448045aeda1efef9bba8c6df4dcf53f4ae6ea9d6638d730b5d not found: ID does not exist" Nov 03 02:19:03.619818 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:03.619789 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g" Nov 03 02:19:03.620241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:03.619862 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt" Nov 03 02:19:04.021012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:04.020984 5463 scope.go:122] "RemoveContainer" containerID="4199f540700f2ed29d2a34f70e3dec7b0e78872698e6bb48e703bd78a790c940" Nov 03 02:19:04.404343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:04.404310 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:04.791256 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:04.791142 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:09.469268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:09.468472 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:12.372962 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:12.372916 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:13.013164 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:19:13.013121 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:19:14.216133 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:14.216096 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:16.832847 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:16.832810 5463 generic.go:356] "Generic (PLEG): container finished" podID="a6d496ad-cf03-4990-9326-a127a83750e2" containerID="2d5583ec3f7631bc8c0ff3725312d77683e20eacfffc051fcba9943bb2b725c2" exitCode=0 Nov 03 02:19:16.833338 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:16.832880 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" event={"ID":"a6d496ad-cf03-4990-9326-a127a83750e2","Type":"ContainerDied","Data":"2d5583ec3f7631bc8c0ff3725312d77683e20eacfffc051fcba9943bb2b725c2"} Nov 03 02:19:16.833338 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:16.833267 5463 scope.go:122] "RemoveContainer" containerID="2d5583ec3f7631bc8c0ff3725312d77683e20eacfffc051fcba9943bb2b725c2" Nov 03 02:19:16.834740 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:16.834713 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:19:16.834834 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:16.834776 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:19:16.834834 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:16.834809 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:19:17.124054 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:17.123968 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:17.842985 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:17.842949 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" event={"ID":"a6d496ad-cf03-4990-9326-a127a83750e2","Type":"ContainerStarted","Data":"a520af000a0d527663206474894eaa3441e97fe2ec4909b281566a865096fceb"} Nov 03 02:19:18.323145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:18.323111 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:19:18.323435 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:18.323411 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 02:19:18.323497 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:18.323463 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" podUID="a6d496ad-cf03-4990-9326-a127a83750e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 02:19:18.771216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:18.767105 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:19.322457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:19.322416 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 02:19:19.322855 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:19.322475 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" podUID="a6d496ad-cf03-4990-9326-a127a83750e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 02:19:19.327508 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:19.327477 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:20.323122 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:20.323078 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 02:19:20.323553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:20.323132 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" podUID="a6d496ad-cf03-4990-9326-a127a83750e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 02:19:21.322868 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:21.322835 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 02:19:21.323056 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:21.322890 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" podUID="a6d496ad-cf03-4990-9326-a127a83750e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 02:19:22.323153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:22.323124 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 02:19:22.323590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:22.323178 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" podUID="a6d496ad-cf03-4990-9326-a127a83750e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 02:19:22.932731 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:22.932694 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:23.327571 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:23.327492 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:19:23.327959 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:23.327874 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:19:23.328549 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:23.328532 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:19:24.033540 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:24.033507 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:24.927020 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:24.926969 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:25.488136 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:25.488106 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc"] Nov 03 02:19:25.598378 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:25.598339 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6"] Nov 03 02:19:25.690532 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:25.690481 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd"] Nov 03 02:19:25.690766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:25.690738 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" podUID="5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" containerName="istio-proxy" containerID="cri-o://f1530ab2c5f087685bd88bb9704620117cd78d4969b531147adfbf0aff897429" gracePeriod=30 Nov 03 02:19:25.693883 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:25.693849 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t"] Nov 03 02:19:25.694063 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:25.694037 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" podUID="0e38d821-9e7d-46a4-93a1-4b4285b1de35" containerName="istio-proxy" containerID="cri-o://ff87290590ce5259684c3b270c0b65b134caaf3c222d9c5bd23b5a982ea5d645" gracePeriod=30 Nov 03 02:19:25.706585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:25.706559 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds"] Nov 03 02:19:25.706730 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:25.706701 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" podUID="6fc604c8-be34-40a7-9fb8-a07424233eb6" containerName="istio-proxy" containerID="cri-o://1de919e861c6dac637ee1dfaf21529240cb2745a3f3a14aba1847a455be02bcb" gracePeriod=30 Nov 03 02:19:25.741059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:25.740982 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt"] Nov 03 02:19:25.741259 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:25.741229 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" podUID="83d426e2-ed63-493c-871e-aa5d62476d77" containerName="istio-proxy" containerID="cri-o://29c0b4e7f860188489ef00c74f9ee87ef484aa0145812bf2d518ffc68627a1df" gracePeriod=30 Nov 03 02:19:25.865417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:25.865379 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:26.022221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:26.022111 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:28.954360 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:28.954324 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:29.116453 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:29.116416 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:29.262645 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:29.262556 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:29.923813 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:29.923779 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qxq6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:19:29.923976 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:29.923868 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" podUID="0e38d821-9e7d-46a4-93a1-4b4285b1de35" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:19:30.116853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:30.116822 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gcscd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:19:30.117272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:30.116887 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" podUID="5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:19:30.280525 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:30.280438 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jlvds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:19:30.280671 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:30.280519 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" podUID="6fc604c8-be34-40a7-9fb8-a07424233eb6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:19:30.488695 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:30.488632 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc" podUID="cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1" containerName="payload-processing" containerID="cri-o://4eac6f7dc6aa6cc0eeab74e359b5b32ff2ed32b403b51d7556ff5fb2315fa635" gracePeriod=25 Nov 03 02:19:30.598694 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:30.598627 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6" podUID="1941d760-2cbe-42ba-a119-d1e4006d3c86" containerName="payload-pre-processing" containerID="cri-o://eb3e3694e1aaae4d863533c0310666e56ab9adac2ec99478edcaa0fc8c4b56ab" gracePeriod=25 Nov 03 02:19:30.908282 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:30.908177 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:30.973811 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:30.973584 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6" Nov 03 02:19:30.977581 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:30.974945 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc" Nov 03 02:19:31.014862 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.014795 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" event={"ID":"6fc604c8-be34-40a7-9fb8-a07424233eb6","Type":"ContainerDied","Data":"1de919e861c6dac637ee1dfaf21529240cb2745a3f3a14aba1847a455be02bcb"} Nov 03 02:19:31.015146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.014748 5463 generic.go:356] "Generic (PLEG): container finished" podID="6fc604c8-be34-40a7-9fb8-a07424233eb6" containerID="1de919e861c6dac637ee1dfaf21529240cb2745a3f3a14aba1847a455be02bcb" exitCode=0 Nov 03 02:19:31.025674 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.025628 5463 generic.go:356] "Generic (PLEG): container finished" podID="cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1" containerID="4eac6f7dc6aa6cc0eeab74e359b5b32ff2ed32b403b51d7556ff5fb2315fa635" exitCode=0 Nov 03 02:19:31.030926 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.027389 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc" event={"ID":"cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1","Type":"ContainerDied","Data":"4eac6f7dc6aa6cc0eeab74e359b5b32ff2ed32b403b51d7556ff5fb2315fa635"} Nov 03 02:19:31.030926 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.027453 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc" event={"ID":"cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1","Type":"ContainerDied","Data":"11a0d7ab6e3f76982f54f19b170c383f72c0a818d9429c10026a49f449d6ca81"} Nov 03 02:19:31.030926 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.027484 5463 scope.go:122] "RemoveContainer" containerID="4eac6f7dc6aa6cc0eeab74e359b5b32ff2ed32b403b51d7556ff5fb2315fa635" Nov 03 02:19:31.030926 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.027646 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc" Nov 03 02:19:31.039321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.036213 5463 generic.go:356] "Generic (PLEG): container finished" podID="83d426e2-ed63-493c-871e-aa5d62476d77" containerID="29c0b4e7f860188489ef00c74f9ee87ef484aa0145812bf2d518ffc68627a1df" exitCode=0 Nov 03 02:19:31.039321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.036394 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" event={"ID":"83d426e2-ed63-493c-871e-aa5d62476d77","Type":"ContainerDied","Data":"29c0b4e7f860188489ef00c74f9ee87ef484aa0145812bf2d518ffc68627a1df"} Nov 03 02:19:31.039321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.038454 5463 generic.go:356] "Generic (PLEG): container finished" podID="0e38d821-9e7d-46a4-93a1-4b4285b1de35" containerID="ff87290590ce5259684c3b270c0b65b134caaf3c222d9c5bd23b5a982ea5d645" exitCode=0 Nov 03 02:19:31.039321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.038535 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" event={"ID":"0e38d821-9e7d-46a4-93a1-4b4285b1de35","Type":"ContainerDied","Data":"ff87290590ce5259684c3b270c0b65b134caaf3c222d9c5bd23b5a982ea5d645"} Nov 03 02:19:31.053771 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.052990 5463 generic.go:356] "Generic (PLEG): container finished" podID="1941d760-2cbe-42ba-a119-d1e4006d3c86" containerID="eb3e3694e1aaae4d863533c0310666e56ab9adac2ec99478edcaa0fc8c4b56ab" exitCode=0 Nov 03 02:19:31.053771 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.053119 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6" event={"ID":"1941d760-2cbe-42ba-a119-d1e4006d3c86","Type":"ContainerDied","Data":"eb3e3694e1aaae4d863533c0310666e56ab9adac2ec99478edcaa0fc8c4b56ab"} Nov 03 02:19:31.053771 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.053165 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6" event={"ID":"1941d760-2cbe-42ba-a119-d1e4006d3c86","Type":"ContainerDied","Data":"d3673b8a9203b0d610631713f16c332bd8c9c9b8bddcc75dabe7b5bad1dd6a1e"} Nov 03 02:19:31.053771 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.053267 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6" Nov 03 02:19:31.059249 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.058287 5463 generic.go:356] "Generic (PLEG): container finished" podID="5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" containerID="f1530ab2c5f087685bd88bb9704620117cd78d4969b531147adfbf0aff897429" exitCode=0 Nov 03 02:19:31.059249 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.058327 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" event={"ID":"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b","Type":"ContainerDied","Data":"f1530ab2c5f087685bd88bb9704620117cd78d4969b531147adfbf0aff897429"} Nov 03 02:19:31.064345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.063059 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1-plugins-config-volume\") pod \"cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1\" (UID: \"cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1\") " Nov 03 02:19:31.064345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.063182 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1-kube-api-access-98flq\" (UniqueName: \"kubernetes.io/projected/cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1-kube-api-access-98flq\") pod \"cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1\" (UID: \"cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1\") " Nov 03 02:19:31.064345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.063258 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1941d760-2cbe-42ba-a119-d1e4006d3c86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1941d760-2cbe-42ba-a119-d1e4006d3c86-plugins-config-volume\") pod \"1941d760-2cbe-42ba-a119-d1e4006d3c86\" (UID: \"1941d760-2cbe-42ba-a119-d1e4006d3c86\") " Nov 03 02:19:31.064345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.063391 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1941d760-2cbe-42ba-a119-d1e4006d3c86-kube-api-access-l4d6g\" (UniqueName: \"kubernetes.io/projected/1941d760-2cbe-42ba-a119-d1e4006d3c86-kube-api-access-l4d6g\") pod \"1941d760-2cbe-42ba-a119-d1e4006d3c86\" (UID: \"1941d760-2cbe-42ba-a119-d1e4006d3c86\") " Nov 03 02:19:31.069447 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.064915 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1941d760-2cbe-42ba-a119-d1e4006d3c86-plugins-config-volume" pod "1941d760-2cbe-42ba-a119-d1e4006d3c86" (UID: "1941d760-2cbe-42ba-a119-d1e4006d3c86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:19:31.069447 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.065107 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1-plugins-config-volume" pod "cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1" (UID: "cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:19:31.069447 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.066522 5463 scope.go:122] "RemoveContainer" containerID="4eac6f7dc6aa6cc0eeab74e359b5b32ff2ed32b403b51d7556ff5fb2315fa635" Nov 03 02:19:31.069447 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:19:31.066906 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eac6f7dc6aa6cc0eeab74e359b5b32ff2ed32b403b51d7556ff5fb2315fa635\": container with ID starting with 4eac6f7dc6aa6cc0eeab74e359b5b32ff2ed32b403b51d7556ff5fb2315fa635 not found: ID does not exist" containerID="4eac6f7dc6aa6cc0eeab74e359b5b32ff2ed32b403b51d7556ff5fb2315fa635" Nov 03 02:19:31.069447 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.066937 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eac6f7dc6aa6cc0eeab74e359b5b32ff2ed32b403b51d7556ff5fb2315fa635"} err="failed to get container status \"4eac6f7dc6aa6cc0eeab74e359b5b32ff2ed32b403b51d7556ff5fb2315fa635\": rpc error: code = NotFound desc = could not find container \"4eac6f7dc6aa6cc0eeab74e359b5b32ff2ed32b403b51d7556ff5fb2315fa635\": container with ID starting with 4eac6f7dc6aa6cc0eeab74e359b5b32ff2ed32b403b51d7556ff5fb2315fa635 not found: ID does not exist" Nov 03 02:19:31.069447 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.066962 5463 scope.go:122] "RemoveContainer" containerID="08d9a0691aa88b6a9ce41589b8d8fb008c907a3d2ae04c1de7d2c8d3a95871df" Nov 03 02:19:31.070413 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.070387 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1-kube-api-access-98flq" pod "cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1" (UID: "cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1"). InnerVolumeSpecName "kube-api-access-98flq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:19:31.073123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.073098 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1941d760-2cbe-42ba-a119-d1e4006d3c86-kube-api-access-l4d6g" pod "1941d760-2cbe-42ba-a119-d1e4006d3c86" (UID: "1941d760-2cbe-42ba-a119-d1e4006d3c86"). InnerVolumeSpecName "kube-api-access-l4d6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:19:31.083263 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.083240 5463 scope.go:122] "RemoveContainer" containerID="eb3e3694e1aaae4d863533c0310666e56ab9adac2ec99478edcaa0fc8c4b56ab" Nov 03 02:19:31.100036 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.100011 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:19:31.104857 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.104771 5463 scope.go:122] "RemoveContainer" containerID="eb3e3694e1aaae4d863533c0310666e56ab9adac2ec99478edcaa0fc8c4b56ab" Nov 03 02:19:31.105364 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:19:31.105292 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb3e3694e1aaae4d863533c0310666e56ab9adac2ec99478edcaa0fc8c4b56ab\": container with ID starting with eb3e3694e1aaae4d863533c0310666e56ab9adac2ec99478edcaa0fc8c4b56ab not found: ID does not exist" containerID="eb3e3694e1aaae4d863533c0310666e56ab9adac2ec99478edcaa0fc8c4b56ab" Nov 03 02:19:31.105364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.105329 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb3e3694e1aaae4d863533c0310666e56ab9adac2ec99478edcaa0fc8c4b56ab"} err="failed to get container status \"eb3e3694e1aaae4d863533c0310666e56ab9adac2ec99478edcaa0fc8c4b56ab\": rpc error: code = NotFound desc = could not find container \"eb3e3694e1aaae4d863533c0310666e56ab9adac2ec99478edcaa0fc8c4b56ab\": container with ID starting with eb3e3694e1aaae4d863533c0310666e56ab9adac2ec99478edcaa0fc8c4b56ab not found: ID does not exist" Nov 03 02:19:31.164502 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.164466 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-workload-certs\") pod \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " Nov 03 02:19:31.165016 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.164751 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-workload-certs" pod "5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" (UID: "5be8748c-8d78-4b2e-86ee-d3198bf9cf3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:31.165016 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.164875 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-workload-socket\") pod \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " Nov 03 02:19:31.165157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.165129 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-workload-socket" pod "5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" (UID: "5be8748c-8d78-4b2e-86ee-d3198bf9cf3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:31.165260 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.165239 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-token\" (UniqueName: \"kubernetes.io/projected/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-token\") pod \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " Nov 03 02:19:31.165368 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.165351 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-podinfo\") pod \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " Nov 03 02:19:31.166791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.165491 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-envoy\") pod \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " Nov 03 02:19:31.166791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.165538 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-data\") pod \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " Nov 03 02:19:31.166791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.165570 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istiod-ca-cert\") pod \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " Nov 03 02:19:31.166791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.165607 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-credential-socket\") pod \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " Nov 03 02:19:31.166791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.165667 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-kube-api-access-nfbpc\" (UniqueName: \"kubernetes.io/projected/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-kube-api-access-nfbpc\") pod \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\" (UID: \"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b\") " Nov 03 02:19:31.166791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.166288 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98flq\" (UniqueName: \"kubernetes.io/projected/cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1-kube-api-access-98flq\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.166791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.166311 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1941d760-2cbe-42ba-a119-d1e4006d3c86-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.166791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.166327 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4d6g\" (UniqueName: \"kubernetes.io/projected/1941d760-2cbe-42ba-a119-d1e4006d3c86-kube-api-access-l4d6g\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.166791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.166342 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.166791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.166355 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.166791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.166369 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.166791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.166649 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istiod-ca-cert" pod "5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" (UID: "5be8748c-8d78-4b2e-86ee-d3198bf9cf3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:19:31.166791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.166712 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-credential-socket" pod "5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" (UID: "5be8748c-8d78-4b2e-86ee-d3198bf9cf3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:31.166791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.166770 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-data" pod "5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" (UID: "5be8748c-8d78-4b2e-86ee-d3198bf9cf3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:31.168525 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.168427 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-token" pod "5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" (UID: "5be8748c-8d78-4b2e-86ee-d3198bf9cf3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:19:31.168775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.168735 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-envoy" pod "5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" (UID: "5be8748c-8d78-4b2e-86ee-d3198bf9cf3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:31.170342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.169646 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-podinfo" pod "5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" (UID: "5be8748c-8d78-4b2e-86ee-d3198bf9cf3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:19:31.172703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.171741 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-kube-api-access-nfbpc" pod "5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" (UID: "5be8748c-8d78-4b2e-86ee-d3198bf9cf3b"). InnerVolumeSpecName "kube-api-access-nfbpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:19:31.174684 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.174601 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:19:31.181507 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.181436 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:19:31.221686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.221665 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 02:19:31.267659 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.267631 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc604c8-be34-40a7-9fb8-a07424233eb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fc604c8-be34-40a7-9fb8-a07424233eb6-istiod-ca-cert\") pod \"6fc604c8-be34-40a7-9fb8-a07424233eb6\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " Nov 03 02:19:31.267795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.267745 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-podinfo\") pod \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " Nov 03 02:19:31.267844 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.267808 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-data\") pod \"6fc604c8-be34-40a7-9fb8-a07424233eb6\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " Nov 03 02:19:31.267885 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.267869 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-token\" (UniqueName: \"kubernetes.io/projected/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-token\") pod \"6fc604c8-be34-40a7-9fb8-a07424233eb6\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " Nov 03 02:19:31.267919 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.267905 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istiod-ca-cert\") pod \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " Nov 03 02:19:31.267954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.267945 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-credential-socket\") pod \"6fc604c8-be34-40a7-9fb8-a07424233eb6\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " Nov 03 02:19:31.268036 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.268018 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc604c8-be34-40a7-9fb8-a07424233eb6-kube-api-access-mnh94\" (UniqueName: \"kubernetes.io/projected/6fc604c8-be34-40a7-9fb8-a07424233eb6-kube-api-access-mnh94\") pod \"6fc604c8-be34-40a7-9fb8-a07424233eb6\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " Nov 03 02:19:31.268068 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.268052 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-token\" (UniqueName: \"kubernetes.io/projected/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-token\") pod \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " Nov 03 02:19:31.268102 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.268087 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-data\") pod \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " Nov 03 02:19:31.268137 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.268120 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-credential-socket\") pod \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " Nov 03 02:19:31.268242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.268225 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-workload-certs\") pod \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " Nov 03 02:19:31.268317 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.268301 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-workload-socket\") pod \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " Nov 03 02:19:31.268376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.268349 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-workload-certs\") pod \"6fc604c8-be34-40a7-9fb8-a07424233eb6\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " Nov 03 02:19:31.268421 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.268378 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-envoy\") pod \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " Nov 03 02:19:31.268455 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.268415 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e38d821-9e7d-46a4-93a1-4b4285b1de35-kube-api-access-s8gpd\" (UniqueName: \"kubernetes.io/projected/0e38d821-9e7d-46a4-93a1-4b4285b1de35-kube-api-access-s8gpd\") pod \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\" (UID: \"0e38d821-9e7d-46a4-93a1-4b4285b1de35\") " Nov 03 02:19:31.268490 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.268456 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-podinfo\") pod \"6fc604c8-be34-40a7-9fb8-a07424233eb6\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " Nov 03 02:19:31.268539 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.268525 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-workload-socket\") pod \"6fc604c8-be34-40a7-9fb8-a07424233eb6\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " Nov 03 02:19:31.268575 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.268564 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-envoy\") pod \"6fc604c8-be34-40a7-9fb8-a07424233eb6\" (UID: \"6fc604c8-be34-40a7-9fb8-a07424233eb6\") " Nov 03 02:19:31.269263 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.269237 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.269263 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.269265 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.269450 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.269283 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.269450 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.269298 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.269450 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.269312 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.269450 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.269326 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfbpc\" (UniqueName: \"kubernetes.io/projected/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-kube-api-access-nfbpc\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.269450 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.269341 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.270530 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.269832 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-data" pod "0e38d821-9e7d-46a4-93a1-4b4285b1de35" (UID: "0e38d821-9e7d-46a4-93a1-4b4285b1de35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:31.270530 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.270174 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-credential-socket" pod "6fc604c8-be34-40a7-9fb8-a07424233eb6" (UID: "6fc604c8-be34-40a7-9fb8-a07424233eb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:31.270530 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.270336 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-data" pod "6fc604c8-be34-40a7-9fb8-a07424233eb6" (UID: "6fc604c8-be34-40a7-9fb8-a07424233eb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:31.270530 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.270522 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-workload-certs" pod "6fc604c8-be34-40a7-9fb8-a07424233eb6" (UID: "6fc604c8-be34-40a7-9fb8-a07424233eb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:31.270844 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.270698 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-credential-socket" pod "0e38d821-9e7d-46a4-93a1-4b4285b1de35" (UID: "0e38d821-9e7d-46a4-93a1-4b4285b1de35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:31.270905 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.270853 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-workload-certs" pod "0e38d821-9e7d-46a4-93a1-4b4285b1de35" (UID: "0e38d821-9e7d-46a4-93a1-4b4285b1de35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:31.271033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.271008 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-workload-socket" pod "0e38d821-9e7d-46a4-93a1-4b4285b1de35" (UID: "0e38d821-9e7d-46a4-93a1-4b4285b1de35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:31.271516 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.271490 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istiod-ca-cert" pod "0e38d821-9e7d-46a4-93a1-4b4285b1de35" (UID: "0e38d821-9e7d-46a4-93a1-4b4285b1de35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:19:31.272160 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.272133 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc604c8-be34-40a7-9fb8-a07424233eb6-istiod-ca-cert" pod "6fc604c8-be34-40a7-9fb8-a07424233eb6" (UID: "6fc604c8-be34-40a7-9fb8-a07424233eb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:19:31.272580 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.272549 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-workload-socket" pod "6fc604c8-be34-40a7-9fb8-a07424233eb6" (UID: "6fc604c8-be34-40a7-9fb8-a07424233eb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:31.273657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.273624 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-token" pod "6fc604c8-be34-40a7-9fb8-a07424233eb6" (UID: "6fc604c8-be34-40a7-9fb8-a07424233eb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:19:31.274122 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.274096 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-envoy" pod "0e38d821-9e7d-46a4-93a1-4b4285b1de35" (UID: "0e38d821-9e7d-46a4-93a1-4b4285b1de35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:31.274681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.274657 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e38d821-9e7d-46a4-93a1-4b4285b1de35-kube-api-access-s8gpd" pod "0e38d821-9e7d-46a4-93a1-4b4285b1de35" (UID: "0e38d821-9e7d-46a4-93a1-4b4285b1de35"). InnerVolumeSpecName "kube-api-access-s8gpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:19:31.275272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.275250 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-podinfo" pod "6fc604c8-be34-40a7-9fb8-a07424233eb6" (UID: "6fc604c8-be34-40a7-9fb8-a07424233eb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:19:31.276500 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.276481 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-token" pod "0e38d821-9e7d-46a4-93a1-4b4285b1de35" (UID: "0e38d821-9e7d-46a4-93a1-4b4285b1de35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:19:31.276568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.276510 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-envoy" pod "6fc604c8-be34-40a7-9fb8-a07424233eb6" (UID: "6fc604c8-be34-40a7-9fb8-a07424233eb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:31.276568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.276517 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc604c8-be34-40a7-9fb8-a07424233eb6-kube-api-access-mnh94" pod "6fc604c8-be34-40a7-9fb8-a07424233eb6" (UID: "6fc604c8-be34-40a7-9fb8-a07424233eb6"). InnerVolumeSpecName "kube-api-access-mnh94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:19:31.277608 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.277581 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-podinfo" pod "0e38d821-9e7d-46a4-93a1-4b4285b1de35" (UID: "0e38d821-9e7d-46a4-93a1-4b4285b1de35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:19:31.370550 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.370516 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-workload-socket\") pod \"83d426e2-ed63-493c-871e-aa5d62476d77\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " Nov 03 02:19:31.371371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.370602 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-istio-data\") pod \"83d426e2-ed63-493c-871e-aa5d62476d77\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " Nov 03 02:19:31.371371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.370654 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-workload-certs\") pod \"83d426e2-ed63-493c-871e-aa5d62476d77\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " Nov 03 02:19:31.371371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.370726 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-istio-envoy\") pod \"83d426e2-ed63-493c-871e-aa5d62476d77\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " Nov 03 02:19:31.371371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.370898 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83d426e2-ed63-493c-871e-aa5d62476d77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83d426e2-ed63-493c-871e-aa5d62476d77-istiod-ca-cert\") pod \"83d426e2-ed63-493c-871e-aa5d62476d77\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " Nov 03 02:19:31.371371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.370927 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83d426e2-ed63-493c-871e-aa5d62476d77-kube-api-access-9h6d9\" (UniqueName: \"kubernetes.io/projected/83d426e2-ed63-493c-871e-aa5d62476d77-kube-api-access-9h6d9\") pod \"83d426e2-ed63-493c-871e-aa5d62476d77\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " Nov 03 02:19:31.371371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.371019 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-credential-socket\") pod \"83d426e2-ed63-493c-871e-aa5d62476d77\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " Nov 03 02:19:31.371371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.371050 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83d426e2-ed63-493c-871e-aa5d62476d77-istio-token\" (UniqueName: \"kubernetes.io/projected/83d426e2-ed63-493c-871e-aa5d62476d77-istio-token\") pod \"83d426e2-ed63-493c-871e-aa5d62476d77\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " Nov 03 02:19:31.371371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.371079 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83d426e2-ed63-493c-871e-aa5d62476d77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83d426e2-ed63-493c-871e-aa5d62476d77-istio-podinfo\") pod \"83d426e2-ed63-493c-871e-aa5d62476d77\" (UID: \"83d426e2-ed63-493c-871e-aa5d62476d77\") " Nov 03 02:19:31.371371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.371123 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-istio-data" pod "83d426e2-ed63-493c-871e-aa5d62476d77" (UID: "83d426e2-ed63-493c-871e-aa5d62476d77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:31.371371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.371313 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-workload-socket" pod "83d426e2-ed63-493c-871e-aa5d62476d77" (UID: "83d426e2-ed63-493c-871e-aa5d62476d77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:31.375431 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.371596 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-workload-certs" pod "83d426e2-ed63-493c-871e-aa5d62476d77" (UID: "83d426e2-ed63-493c-871e-aa5d62476d77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:31.375431 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.371774 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-credential-socket" pod "83d426e2-ed63-493c-871e-aa5d62476d77" (UID: "83d426e2-ed63-493c-871e-aa5d62476d77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:31.375431 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.371819 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83d426e2-ed63-493c-871e-aa5d62476d77-istiod-ca-cert" pod "83d426e2-ed63-493c-871e-aa5d62476d77" (UID: "83d426e2-ed63-493c-871e-aa5d62476d77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:19:31.375431 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.373896 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-istio-envoy" pod "83d426e2-ed63-493c-871e-aa5d62476d77" (UID: "83d426e2-ed63-493c-871e-aa5d62476d77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:19:31.375431 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.374675 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83d426e2-ed63-493c-871e-aa5d62476d77-istio-podinfo" pod "83d426e2-ed63-493c-871e-aa5d62476d77" (UID: "83d426e2-ed63-493c-871e-aa5d62476d77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:19:31.375684 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.375598 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d426e2-ed63-493c-871e-aa5d62476d77-istio-token" pod "83d426e2-ed63-493c-871e-aa5d62476d77" (UID: "83d426e2-ed63-493c-871e-aa5d62476d77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:19:31.376687 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376663 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnh94\" (UniqueName: \"kubernetes.io/projected/6fc604c8-be34-40a7-9fb8-a07424233eb6-kube-api-access-mnh94\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.376792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376693 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.376792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376708 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.376792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376721 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.376792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376734 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83d426e2-ed63-493c-871e-aa5d62476d77-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.376792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376747 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.376792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376761 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.376792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376775 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83d426e2-ed63-493c-871e-aa5d62476d77-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.376792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376788 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83d426e2-ed63-493c-871e-aa5d62476d77-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.377248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376801 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.377248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376833 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.377248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376847 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.377248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376862 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.377248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376875 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8gpd\" (UniqueName: \"kubernetes.io/projected/0e38d821-9e7d-46a4-93a1-4b4285b1de35-kube-api-access-s8gpd\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.377248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376889 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.377248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376902 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.377248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376915 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.377248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376928 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.377248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376940 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.377248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376953 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fc604c8-be34-40a7-9fb8-a07424233eb6-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.377248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376966 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83d426e2-ed63-493c-871e-aa5d62476d77-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.377248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376979 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.377248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.376992 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.377248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.377004 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fc604c8-be34-40a7-9fb8-a07424233eb6-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.377248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.377017 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e38d821-9e7d-46a4-93a1-4b4285b1de35-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.377248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.377030 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fc604c8-be34-40a7-9fb8-a07424233eb6-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.379143 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.379121 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d426e2-ed63-493c-871e-aa5d62476d77-kube-api-access-9h6d9" pod "83d426e2-ed63-493c-871e-aa5d62476d77" (UID: "83d426e2-ed63-493c-871e-aa5d62476d77"). InnerVolumeSpecName "kube-api-access-9h6d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:19:31.482001 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.481909 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h6d9\" (UniqueName: \"kubernetes.io/projected/83d426e2-ed63-493c-871e-aa5d62476d77-kube-api-access-9h6d9\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:31.835548 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.835477 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6"] Nov 03 02:19:31.840401 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.840376 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6198c6-85556f46c-cw9r6"] Nov 03 02:19:31.950787 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.950753 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc"] Nov 03 02:19:31.955897 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:31.955867 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6198c6-b5fc9f69-rm8cc"] Nov 03 02:19:32.066959 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.066926 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" Nov 03 02:19:32.067129 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.066932 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t" event={"ID":"0e38d821-9e7d-46a4-93a1-4b4285b1de35","Type":"ContainerDied","Data":"a4d95db0617707dd54efa40df0f3f5051461383756d0c006adf87b79180083f7"} Nov 03 02:19:32.067129 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.067084 5463 scope.go:122] "RemoveContainer" containerID="ff87290590ce5259684c3b270c0b65b134caaf3c222d9c5bd23b5a982ea5d645" Nov 03 02:19:32.069795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.069776 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" Nov 03 02:19:32.069929 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.069791 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd" event={"ID":"5be8748c-8d78-4b2e-86ee-d3198bf9cf3b","Type":"ContainerDied","Data":"3b3da3fe43224b17ea867897ae4e402d1eb062b596c441c1d922047ea1f52d22"} Nov 03 02:19:32.071455 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.071427 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" Nov 03 02:19:32.071568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.071475 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds" event={"ID":"6fc604c8-be34-40a7-9fb8-a07424233eb6","Type":"ContainerDied","Data":"6ad69ab03408df4865bdac5a08acd56762c0ceb802f2da9306d117795ea05aba"} Nov 03 02:19:32.075582 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.075552 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" Nov 03 02:19:32.075582 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.075568 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt" event={"ID":"83d426e2-ed63-493c-871e-aa5d62476d77","Type":"ContainerDied","Data":"f8a8eede948885722349c5e99b6c0359dd2ae614fbc54e8d477c07d8a1d3f00a"} Nov 03 02:19:32.080632 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.080614 5463 scope.go:122] "RemoveContainer" containerID="f1530ab2c5f087685bd88bb9704620117cd78d4969b531147adfbf0aff897429" Nov 03 02:19:32.091107 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.091090 5463 scope.go:122] "RemoveContainer" containerID="1de919e861c6dac637ee1dfaf21529240cb2745a3f3a14aba1847a455be02bcb" Nov 03 02:19:32.100212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.100177 5463 scope.go:122] "RemoveContainer" containerID="29c0b4e7f860188489ef00c74f9ee87ef484aa0145812bf2d518ffc68627a1df" Nov 03 02:19:32.233314 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.233283 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd"] Nov 03 02:19:32.236630 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.236609 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gcscd"] Nov 03 02:19:32.346898 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.346821 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t"] Nov 03 02:19:32.351012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.350991 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qxq6t"] Nov 03 02:19:32.450717 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.450680 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e38d821-9e7d-46a4-93a1-4b4285b1de35" path="/var/lib/kubelet/pods/0e38d821-9e7d-46a4-93a1-4b4285b1de35/volumes" Nov 03 02:19:32.451380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.451354 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1941d760-2cbe-42ba-a119-d1e4006d3c86" path="/var/lib/kubelet/pods/1941d760-2cbe-42ba-a119-d1e4006d3c86/volumes" Nov 03 02:19:32.451990 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.451972 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" path="/var/lib/kubelet/pods/5be8748c-8d78-4b2e-86ee-d3198bf9cf3b/volumes" Nov 03 02:19:32.452925 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.452906 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1" path="/var/lib/kubelet/pods/cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1/volumes" Nov 03 02:19:32.466169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.465960 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds"] Nov 03 02:19:32.482360 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.482330 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jlvds"] Nov 03 02:19:32.583618 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.583578 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt"] Nov 03 02:19:32.586994 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:32.586963 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m6hqt"] Nov 03 02:19:34.155323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:34.155269 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:34.446452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:34.446380 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc604c8-be34-40a7-9fb8-a07424233eb6" path="/var/lib/kubelet/pods/6fc604c8-be34-40a7-9fb8-a07424233eb6/volumes" Nov 03 02:19:34.446870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:34.446855 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83d426e2-ed63-493c-871e-aa5d62476d77" path="/var/lib/kubelet/pods/83d426e2-ed63-493c-871e-aa5d62476d77/volumes" Nov 03 02:19:38.650156 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.650122 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4"] Nov 03 02:19:38.650781 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.650765 5463 cpu_manager.go:414] "containerMap: removing container" podUID="6fc604c8-be34-40a7-9fb8-a07424233eb6" containerName="istio-proxy" Nov 03 02:19:38.650781 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.650780 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc604c8-be34-40a7-9fb8-a07424233eb6" containerName="istio-proxy" Nov 03 02:19:38.650908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.650792 5463 cpu_manager.go:414] "containerMap: removing container" podUID="7ead49c5-780b-498a-84ab-bbbadea4d8b7" containerName="test-curl-7241-781b67" Nov 03 02:19:38.650908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.650797 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ead49c5-780b-498a-84ab-bbbadea4d8b7" containerName="test-curl-7241-781b67" Nov 03 02:19:38.650908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.650814 5463 cpu_manager.go:414] "containerMap: removing container" podUID="cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1" containerName="payload-processing" Nov 03 02:19:38.650908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.650819 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1" containerName="payload-processing" Nov 03 02:19:38.650908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.650835 5463 cpu_manager.go:414] "containerMap: removing container" podUID="83d426e2-ed63-493c-871e-aa5d62476d77" containerName="istio-proxy" Nov 03 02:19:38.650908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.650840 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83d426e2-ed63-493c-871e-aa5d62476d77" containerName="istio-proxy" Nov 03 02:19:38.650908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.650851 5463 cpu_manager.go:414] "containerMap: removing container" podUID="83d426e2-ed63-493c-871e-aa5d62476d77" containerName="istio-proxy" Nov 03 02:19:38.650908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.650859 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83d426e2-ed63-493c-871e-aa5d62476d77" containerName="istio-proxy" Nov 03 02:19:38.650908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.650868 5463 cpu_manager.go:414] "containerMap: removing container" podUID="1941d760-2cbe-42ba-a119-d1e4006d3c86" containerName="payload-pre-processing" Nov 03 02:19:38.650908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.650873 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1941d760-2cbe-42ba-a119-d1e4006d3c86" containerName="payload-pre-processing" Nov 03 02:19:38.650908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.650892 5463 cpu_manager.go:414] "containerMap: removing container" podUID="5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" containerName="istio-proxy" Nov 03 02:19:38.650908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.650896 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" containerName="istio-proxy" Nov 03 02:19:38.650908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.650909 5463 cpu_manager.go:414] "containerMap: removing container" podUID="0e38d821-9e7d-46a4-93a1-4b4285b1de35" containerName="istio-proxy" Nov 03 02:19:38.650908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.650914 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e38d821-9e7d-46a4-93a1-4b4285b1de35" containerName="istio-proxy" Nov 03 02:19:38.651642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.650994 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e38d821-9e7d-46a4-93a1-4b4285b1de35" containerName="istio-proxy" Nov 03 02:19:38.651642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.651005 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc604c8-be34-40a7-9fb8-a07424233eb6" containerName="istio-proxy" Nov 03 02:19:38.651642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.651011 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ead49c5-780b-498a-84ab-bbbadea4d8b7" containerName="test-curl-7241-781b67" Nov 03 02:19:38.651642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.651020 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb37d5d5-ece9-4fc1-8c06-62e85a3e1ec1" containerName="payload-processing" Nov 03 02:19:38.651642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.651028 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="83d426e2-ed63-493c-871e-aa5d62476d77" containerName="istio-proxy" Nov 03 02:19:38.651642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.651034 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="83d426e2-ed63-493c-871e-aa5d62476d77" containerName="istio-proxy" Nov 03 02:19:38.651642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.651041 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="1941d760-2cbe-42ba-a119-d1e4006d3c86" containerName="payload-pre-processing" Nov 03 02:19:38.651642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.651049 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="5be8748c-8d78-4b2e-86ee-d3198bf9cf3b" containerName="istio-proxy" Nov 03 02:19:38.656657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.656637 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" Nov 03 02:19:38.672997 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.672970 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7130486e-0af7-40d3-8279-695ee3927707-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4\" (UID: \"7130486e-0af7-40d3-8279-695ee3927707\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" Nov 03 02:19:38.673118 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.673030 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7130486e-0af7-40d3-8279-695ee3927707-metrics-certs\") pod \"maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4\" (UID: \"7130486e-0af7-40d3-8279-695ee3927707\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" Nov 03 02:19:38.673118 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.673110 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7130486e-0af7-40d3-8279-695ee3927707-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4\" (UID: \"7130486e-0af7-40d3-8279-695ee3927707\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" Nov 03 02:19:38.673298 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.673278 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4xxx\" (UniqueName: \"kubernetes.io/projected/7130486e-0af7-40d3-8279-695ee3927707-kube-api-access-s4xxx\") pod \"maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4\" (UID: \"7130486e-0af7-40d3-8279-695ee3927707\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" Nov 03 02:19:38.706727 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.706701 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4"] Nov 03 02:19:38.753204 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.753160 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-b3bbbc-serving-cert\"" Nov 03 02:19:38.774889 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.774857 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4xxx\" (UniqueName: \"kubernetes.io/projected/7130486e-0af7-40d3-8279-695ee3927707-kube-api-access-s4xxx\") pod \"maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4\" (UID: \"7130486e-0af7-40d3-8279-695ee3927707\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" Nov 03 02:19:38.775056 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.774949 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7130486e-0af7-40d3-8279-695ee3927707-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4\" (UID: \"7130486e-0af7-40d3-8279-695ee3927707\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" Nov 03 02:19:38.775056 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.775010 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7130486e-0af7-40d3-8279-695ee3927707-metrics-certs\") pod \"maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4\" (UID: \"7130486e-0af7-40d3-8279-695ee3927707\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" Nov 03 02:19:38.775178 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.775062 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7130486e-0af7-40d3-8279-695ee3927707-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4\" (UID: \"7130486e-0af7-40d3-8279-695ee3927707\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" Nov 03 02:19:38.775275 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:19:38.775247 5463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-serving-cert: secret "maas-api-e2e-ipp-a-b3bbbc-serving-cert" not found Nov 03 02:19:38.775408 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:19:38.775320 5463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7130486e-0af7-40d3-8279-695ee3927707-maas-api-tls podName:7130486e-0af7-40d3-8279-695ee3927707 nodeName:}" failed. No retries permitted until 2026-11-03 02:19:39.275297514 +0000 UTC m=+1660.959406653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7130486e-0af7-40d3-8279-695ee3927707-maas-api-tls") pod "maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" (UID: "7130486e-0af7-40d3-8279-695ee3927707") : secret "maas-api-e2e-ipp-a-b3bbbc-serving-cert" not found Nov 03 02:19:38.775777 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.775753 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7130486e-0af7-40d3-8279-695ee3927707-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4\" (UID: \"7130486e-0af7-40d3-8279-695ee3927707\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" Nov 03 02:19:38.777655 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.777626 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7130486e-0af7-40d3-8279-695ee3927707-metrics-certs\") pod \"maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4\" (UID: \"7130486e-0af7-40d3-8279-695ee3927707\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" Nov 03 02:19:38.839034 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.839003 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4xxx\" (UniqueName: \"kubernetes.io/projected/7130486e-0af7-40d3-8279-695ee3927707-kube-api-access-s4xxx\") pod \"maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4\" (UID: \"7130486e-0af7-40d3-8279-695ee3927707\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" Nov 03 02:19:38.865076 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:38.865049 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:39.019593 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:39.019508 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:39.175775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:39.175737 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:39.282782 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:39.282689 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7130486e-0af7-40d3-8279-695ee3927707-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4\" (UID: \"7130486e-0af7-40d3-8279-695ee3927707\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" Nov 03 02:19:39.285386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:39.285357 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7130486e-0af7-40d3-8279-695ee3927707-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4\" (UID: \"7130486e-0af7-40d3-8279-695ee3927707\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" Nov 03 02:19:39.571247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:39.571151 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" Nov 03 02:19:39.945301 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:39.945270 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4"] Nov 03 02:19:39.946366 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:19:39.946338 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7130486e_0af7_40d3_8279_695ee3927707.slice/crio-8ac4599d44b7854f3b6e035603ca3655664199d5dddfdaa4fc82519fd2e43f4b WatchSource:0}: Error finding container 8ac4599d44b7854f3b6e035603ca3655664199d5dddfdaa4fc82519fd2e43f4b: Status 404 returned error can't find the container with id 8ac4599d44b7854f3b6e035603ca3655664199d5dddfdaa4fc82519fd2e43f4b Nov 03 02:19:40.136152 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:40.136119 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" event={"ID":"7130486e-0af7-40d3-8279-695ee3927707","Type":"ContainerStarted","Data":"8ac4599d44b7854f3b6e035603ca3655664199d5dddfdaa4fc82519fd2e43f4b"} Nov 03 02:19:41.151768 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:41.151725 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" event={"ID":"7130486e-0af7-40d3-8279-695ee3927707","Type":"ContainerStarted","Data":"fec4725423dec6f1b0d186e1511dad72aaea4db2b42e19a824d93ed371bfc47e"} Nov 03 02:19:41.152861 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:41.152829 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" Nov 03 02:19:41.526181 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:41.526090 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:41.949369 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:41.949299 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:43.003932 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:43.003867 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" podStartSLOduration=4.158026798 podStartE2EDuration="5.003847318s" totalImagesPullingTime="845.82052ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:19:41.270434481 +0000 UTC m=+1662.954543640" watchObservedRunningTime="2026-11-03 02:19:43.003847318 +0000 UTC m=+1664.687956474" Nov 03 02:19:43.007906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:43.007875 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p"] Nov 03 02:19:43.013234 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:19:43.012428 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:19:43.115362 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:43.115330 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf"] Nov 03 02:19:43.489220 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:43.488480 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:43.538846 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:43.538814 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2"] Nov 03 02:19:43.539019 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:43.538999 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" podUID="698b1c2c-aed3-496c-9a2d-dbe3137090f6" containerName="maas-api" containerID="cri-o://1cc007583eca8acf71f9e35415bc525090e63dbad614a48070435a98f74d8a3f" gracePeriod=30 Nov 03 02:19:43.675136 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:43.674116 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:43.824804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:43.824541 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:43.916655 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:43.916624 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" Nov 03 02:19:43.983164 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:43.983133 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/698b1c2c-aed3-496c-9a2d-dbe3137090f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/698b1c2c-aed3-496c-9a2d-dbe3137090f6-metrics-certs\") pod \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\" (UID: \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\") " Nov 03 02:19:43.983358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:43.983264 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/698b1c2c-aed3-496c-9a2d-dbe3137090f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/698b1c2c-aed3-496c-9a2d-dbe3137090f6-maas-api-tls\") pod \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\" (UID: \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\") " Nov 03 02:19:43.983358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:43.983327 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/698b1c2c-aed3-496c-9a2d-dbe3137090f6-kube-api-access-xgj5w\" (UniqueName: \"kubernetes.io/projected/698b1c2c-aed3-496c-9a2d-dbe3137090f6-kube-api-access-xgj5w\") pod \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\" (UID: \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\") " Nov 03 02:19:43.983564 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:43.983547 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/698b1c2c-aed3-496c-9a2d-dbe3137090f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/698b1c2c-aed3-496c-9a2d-dbe3137090f6-openshift-service-ca\") pod \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\" (UID: \"698b1c2c-aed3-496c-9a2d-dbe3137090f6\") " Nov 03 02:19:43.983998 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:43.983970 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/698b1c2c-aed3-496c-9a2d-dbe3137090f6-openshift-service-ca" pod "698b1c2c-aed3-496c-9a2d-dbe3137090f6" (UID: "698b1c2c-aed3-496c-9a2d-dbe3137090f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:19:43.984134 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:43.984108 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/698b1c2c-aed3-496c-9a2d-dbe3137090f6-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:43.985597 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:43.985575 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/698b1c2c-aed3-496c-9a2d-dbe3137090f6-metrics-certs" pod "698b1c2c-aed3-496c-9a2d-dbe3137090f6" (UID: "698b1c2c-aed3-496c-9a2d-dbe3137090f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:19:43.985725 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:43.985705 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698b1c2c-aed3-496c-9a2d-dbe3137090f6-kube-api-access-xgj5w" pod "698b1c2c-aed3-496c-9a2d-dbe3137090f6" (UID: "698b1c2c-aed3-496c-9a2d-dbe3137090f6"). InnerVolumeSpecName "kube-api-access-xgj5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:19:43.986538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:43.986518 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/698b1c2c-aed3-496c-9a2d-dbe3137090f6-maas-api-tls" pod "698b1c2c-aed3-496c-9a2d-dbe3137090f6" (UID: "698b1c2c-aed3-496c-9a2d-dbe3137090f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:19:44.086241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:44.085673 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/698b1c2c-aed3-496c-9a2d-dbe3137090f6-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:44.086241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:44.085712 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgj5w\" (UniqueName: \"kubernetes.io/projected/698b1c2c-aed3-496c-9a2d-dbe3137090f6-kube-api-access-xgj5w\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:44.086241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:44.085729 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698b1c2c-aed3-496c-9a2d-dbe3137090f6-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:44.225970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:44.225929 5463 generic.go:356] "Generic (PLEG): container finished" podID="698b1c2c-aed3-496c-9a2d-dbe3137090f6" containerID="1cc007583eca8acf71f9e35415bc525090e63dbad614a48070435a98f74d8a3f" exitCode=0 Nov 03 02:19:44.226165 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:44.226019 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" Nov 03 02:19:44.226165 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:44.226025 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" event={"ID":"698b1c2c-aed3-496c-9a2d-dbe3137090f6","Type":"ContainerDied","Data":"1cc007583eca8acf71f9e35415bc525090e63dbad614a48070435a98f74d8a3f"} Nov 03 02:19:44.226165 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:44.226092 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2" event={"ID":"698b1c2c-aed3-496c-9a2d-dbe3137090f6","Type":"ContainerDied","Data":"18e708b0d85c6f7f456bf4d5ad005b63e5d17cf82604981bcdf030f0a469edd2"} Nov 03 02:19:44.226165 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:44.226125 5463 scope.go:122] "RemoveContainer" containerID="1cc007583eca8acf71f9e35415bc525090e63dbad614a48070435a98f74d8a3f" Nov 03 02:19:44.240404 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:44.240370 5463 scope.go:122] "RemoveContainer" containerID="1cc007583eca8acf71f9e35415bc525090e63dbad614a48070435a98f74d8a3f" Nov 03 02:19:44.240677 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:19:44.240658 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cc007583eca8acf71f9e35415bc525090e63dbad614a48070435a98f74d8a3f\": container with ID starting with 1cc007583eca8acf71f9e35415bc525090e63dbad614a48070435a98f74d8a3f not found: ID does not exist" containerID="1cc007583eca8acf71f9e35415bc525090e63dbad614a48070435a98f74d8a3f" Nov 03 02:19:44.240764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:44.240682 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cc007583eca8acf71f9e35415bc525090e63dbad614a48070435a98f74d8a3f"} err="failed to get container status \"1cc007583eca8acf71f9e35415bc525090e63dbad614a48070435a98f74d8a3f\": rpc error: code = NotFound desc = could not find container \"1cc007583eca8acf71f9e35415bc525090e63dbad614a48070435a98f74d8a3f\": container with ID starting with 1cc007583eca8acf71f9e35415bc525090e63dbad614a48070435a98f74d8a3f not found: ID does not exist" Nov 03 02:19:44.397292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:44.397255 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2"] Nov 03 02:19:44.406770 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:44.406738 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-tbsr2"] Nov 03 02:19:44.445823 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:44.445791 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="698b1c2c-aed3-496c-9a2d-dbe3137090f6" path="/var/lib/kubelet/pods/698b1c2c-aed3-496c-9a2d-dbe3137090f6/volumes" Nov 03 02:19:44.543411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:44.543366 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:45.031265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.031233 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d"] Nov 03 02:19:45.031869 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.031856 5463 cpu_manager.go:414] "containerMap: removing container" podUID="698b1c2c-aed3-496c-9a2d-dbe3137090f6" containerName="maas-api" Nov 03 02:19:45.031915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.031870 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="698b1c2c-aed3-496c-9a2d-dbe3137090f6" containerName="maas-api" Nov 03 02:19:45.032010 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.032001 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="698b1c2c-aed3-496c-9a2d-dbe3137090f6" containerName="maas-api" Nov 03 02:19:45.038171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.038141 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.115213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.115067 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d"] Nov 03 02:19:45.146075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.117490 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:45.146075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.141487 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5ef2c8-openshift-default-dockercfg-zlfxw\"" Nov 03 02:19:45.227110 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.226861 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-workload-socket\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.227110 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.226963 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-envoy\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.228464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.227143 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-workload-certs\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.228464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.227219 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-data\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.228464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.227362 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fwft\" (UniqueName: \"kubernetes.io/projected/ba02d63a-95b8-4195-b07d-4f0e68398761-kube-api-access-2fwft\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.228464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.227534 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba02d63a-95b8-4195-b07d-4f0e68398761-istiod-ca-cert\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.228464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.227565 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-podinfo\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.228464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.227635 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-token\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.228464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.227858 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-credential-socket\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.321965 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.321888 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2"] Nov 03 02:19:45.329146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.329117 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-credential-socket\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.329308 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.329230 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-workload-socket\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.329308 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.329269 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-envoy\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.329390 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.329340 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-workload-certs\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.329390 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.329376 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-data\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.329457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.329417 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fwft\" (UniqueName: \"kubernetes.io/projected/ba02d63a-95b8-4195-b07d-4f0e68398761-kube-api-access-2fwft\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.329531 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.329515 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba02d63a-95b8-4195-b07d-4f0e68398761-istiod-ca-cert\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.329587 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.329551 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-podinfo\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.329587 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.329570 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-credential-socket\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.329587 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.329579 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-token\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.329747 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.329585 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-workload-socket\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.329970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.329948 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-data\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.330156 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.330136 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-workload-certs\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.330275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.330211 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba02d63a-95b8-4195-b07d-4f0e68398761-istiod-ca-cert\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.331819 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.331802 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-envoy\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.332808 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.332787 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-podinfo\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.386393 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.386361 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-token\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.386550 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.386369 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fwft\" (UniqueName: \"kubernetes.io/projected/ba02d63a-95b8-4195-b07d-4f0e68398761-kube-api-access-2fwft\") pod \"e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.433379 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.433345 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9"] Nov 03 02:19:45.439322 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.439287 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:45.605995 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.605697 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:45.794244 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.783660 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:45.929717 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.929684 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:45.933958 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:19:45.933927 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba02d63a_95b8_4195_b07d_4f0e68398761.slice/crio-e8df50909b83dc0b4f3d39b57d60192212e923e5ac62f1e69fec59973a3d3701 WatchSource:0}: Error finding container e8df50909b83dc0b4f3d39b57d60192212e923e5ac62f1e69fec59973a3d3701: Status 404 returned error can't find the container with id e8df50909b83dc0b4f3d39b57d60192212e923e5ac62f1e69fec59973a3d3701 Nov 03 02:19:45.934204 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.934177 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d"] Nov 03 02:19:45.936462 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.936431 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:19:45.936514 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.936496 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:19:45.936552 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:45.936524 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:19:46.264272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:46.261454 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" event={"ID":"ba02d63a-95b8-4195-b07d-4f0e68398761","Type":"ContainerStarted","Data":"66b1b195d206e7e6365a1b490e570eba02f228ca6191131438bfde3c3e70e576"} Nov 03 02:19:46.264272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:46.261523 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" event={"ID":"ba02d63a-95b8-4195-b07d-4f0e68398761","Type":"ContainerStarted","Data":"e8df50909b83dc0b4f3d39b57d60192212e923e5ac62f1e69fec59973a3d3701"} Nov 03 02:19:46.389242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:46.388546 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" podStartSLOduration=2.388526614 podStartE2EDuration="2.388526614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:19:46.379538727 +0000 UTC m=+1668.063647882" watchObservedRunningTime="2026-11-03 02:19:46.388526614 +0000 UTC m=+1668.072635770" Nov 03 02:19:46.445250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:46.445222 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:47.189475 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:47.189431 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" Nov 03 02:19:47.445352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:47.445275 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:47.679879 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:47.679849 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:48.008961 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.008915 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p" podUID="bd79820a-5bdb-449a-9e7f-559755e6e9ee" containerName="payload-processing" containerID="cri-o://90dcccbca5425e9820bad1eff04179c7c0baf09ee59dd793280f9198c94d8144" gracePeriod=25 Nov 03 02:19:48.116653 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.116599 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf" podUID="c6a22f93-19ab-4222-9f41-9ad94cc2642c" containerName="payload-pre-processing" containerID="cri-o://452106c15460077f1f55472c31392b1fcf532e8e8420f59659fd736dff6e7ce0" gracePeriod=25 Nov 03 02:19:48.291994 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.291917 5463 generic.go:356] "Generic (PLEG): container finished" podID="bd79820a-5bdb-449a-9e7f-559755e6e9ee" containerID="90dcccbca5425e9820bad1eff04179c7c0baf09ee59dd793280f9198c94d8144" exitCode=0 Nov 03 02:19:48.292161 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.292075 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p" event={"ID":"bd79820a-5bdb-449a-9e7f-559755e6e9ee","Type":"ContainerDied","Data":"90dcccbca5425e9820bad1eff04179c7c0baf09ee59dd793280f9198c94d8144"} Nov 03 02:19:48.294940 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.294911 5463 generic.go:356] "Generic (PLEG): container finished" podID="c6a22f93-19ab-4222-9f41-9ad94cc2642c" containerID="452106c15460077f1f55472c31392b1fcf532e8e8420f59659fd736dff6e7ce0" exitCode=0 Nov 03 02:19:48.296257 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.296232 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf" event={"ID":"c6a22f93-19ab-4222-9f41-9ad94cc2642c","Type":"ContainerDied","Data":"452106c15460077f1f55472c31392b1fcf532e8e8420f59659fd736dff6e7ce0"} Nov 03 02:19:48.296350 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.296307 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:48.300529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.300498 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:19:48.329448 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.329422 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p" Nov 03 02:19:48.401376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.401349 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd79820a-5bdb-449a-9e7f-559755e6e9ee-kube-api-access-msslv\" (UniqueName: \"kubernetes.io/projected/bd79820a-5bdb-449a-9e7f-559755e6e9ee-kube-api-access-msslv\") pod \"bd79820a-5bdb-449a-9e7f-559755e6e9ee\" (UID: \"bd79820a-5bdb-449a-9e7f-559755e6e9ee\") " Nov 03 02:19:48.401518 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.401424 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd79820a-5bdb-449a-9e7f-559755e6e9ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd79820a-5bdb-449a-9e7f-559755e6e9ee-plugins-config-volume\") pod \"bd79820a-5bdb-449a-9e7f-559755e6e9ee\" (UID: \"bd79820a-5bdb-449a-9e7f-559755e6e9ee\") " Nov 03 02:19:48.401827 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.401794 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd79820a-5bdb-449a-9e7f-559755e6e9ee-plugins-config-volume" pod "bd79820a-5bdb-449a-9e7f-559755e6e9ee" (UID: "bd79820a-5bdb-449a-9e7f-559755e6e9ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:19:48.402175 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.402154 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd79820a-5bdb-449a-9e7f-559755e6e9ee-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:48.403905 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.403883 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd79820a-5bdb-449a-9e7f-559755e6e9ee-kube-api-access-msslv" pod "bd79820a-5bdb-449a-9e7f-559755e6e9ee" (UID: "bd79820a-5bdb-449a-9e7f-559755e6e9ee"). InnerVolumeSpecName "kube-api-access-msslv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:19:48.409144 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.409123 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf" Nov 03 02:19:48.503453 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.503422 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a22f93-19ab-4222-9f41-9ad94cc2642c-kube-api-access-29bht\" (UniqueName: \"kubernetes.io/projected/c6a22f93-19ab-4222-9f41-9ad94cc2642c-kube-api-access-29bht\") pod \"c6a22f93-19ab-4222-9f41-9ad94cc2642c\" (UID: \"c6a22f93-19ab-4222-9f41-9ad94cc2642c\") " Nov 03 02:19:48.503905 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.503671 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6a22f93-19ab-4222-9f41-9ad94cc2642c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a22f93-19ab-4222-9f41-9ad94cc2642c-plugins-config-volume\") pod \"c6a22f93-19ab-4222-9f41-9ad94cc2642c\" (UID: \"c6a22f93-19ab-4222-9f41-9ad94cc2642c\") " Nov 03 02:19:48.504255 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.504232 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a22f93-19ab-4222-9f41-9ad94cc2642c-plugins-config-volume" pod "c6a22f93-19ab-4222-9f41-9ad94cc2642c" (UID: "c6a22f93-19ab-4222-9f41-9ad94cc2642c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:19:48.504482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.504337 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msslv\" (UniqueName: \"kubernetes.io/projected/bd79820a-5bdb-449a-9e7f-559755e6e9ee-kube-api-access-msslv\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:48.504482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.504358 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a22f93-19ab-4222-9f41-9ad94cc2642c-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:48.506544 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.506512 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a22f93-19ab-4222-9f41-9ad94cc2642c-kube-api-access-29bht" pod "c6a22f93-19ab-4222-9f41-9ad94cc2642c" (UID: "c6a22f93-19ab-4222-9f41-9ad94cc2642c"). InnerVolumeSpecName "kube-api-access-29bht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:19:48.605960 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:48.605879 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29bht\" (UniqueName: \"kubernetes.io/projected/c6a22f93-19ab-4222-9f41-9ad94cc2642c-kube-api-access-29bht\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:49.205558 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:49.205516 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:49.307433 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:49.306558 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf" event={"ID":"c6a22f93-19ab-4222-9f41-9ad94cc2642c","Type":"ContainerDied","Data":"300e51bf46a42546a66ad7d22db1085a30e8347249dcd4c63b2a9eab8d3cf241"} Nov 03 02:19:49.307433 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:49.306629 5463 scope.go:122] "RemoveContainer" containerID="452106c15460077f1f55472c31392b1fcf532e8e8420f59659fd736dff6e7ce0" Nov 03 02:19:49.307433 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:49.306781 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf" Nov 03 02:19:49.309522 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:49.309260 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p" event={"ID":"bd79820a-5bdb-449a-9e7f-559755e6e9ee","Type":"ContainerDied","Data":"56f8cfefb5ea67fe202b52732cfafc2c2afe10ee942beffd55135a21c9de35be"} Nov 03 02:19:49.309522 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:49.309319 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p" Nov 03 02:19:49.320427 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:49.320409 5463 scope.go:122] "RemoveContainer" containerID="90dcccbca5425e9820bad1eff04179c7c0baf09ee59dd793280f9198c94d8144" Nov 03 02:19:49.369674 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:49.369263 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:49.481057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:49.481022 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf"] Nov 03 02:19:49.521955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:49.521926 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:49.530815 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:49.530791 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-t6nhf"] Nov 03 02:19:49.642004 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:49.641909 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p"] Nov 03 02:19:49.644946 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:49.644920 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-qns7p"] Nov 03 02:19:50.129718 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:50.129683 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:50.323162 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:50.323125 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2" podUID="8fbbc88e-2111-44d1-a28d-537cc9aa1094" containerName="payload-processing" containerID="cri-o://eaf3617fcfdddabfa5740500b8149d8aaf0896cc9e34ff6395f79d061d46ef74" gracePeriod=25 Nov 03 02:19:50.435553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:50.433868 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9" podUID="cd144ab9-c755-4886-a81f-cbd747a5e5c3" containerName="payload-pre-processing" containerID="cri-o://7fb5872e8eaf0cf2079899b3d17f65f51a6061964249621a46e59128c8f1a234" gracePeriod=25 Nov 03 02:19:50.450769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:50.450677 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd79820a-5bdb-449a-9e7f-559755e6e9ee" path="/var/lib/kubelet/pods/bd79820a-5bdb-449a-9e7f-559755e6e9ee/volumes" Nov 03 02:19:50.451138 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:50.451123 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6a22f93-19ab-4222-9f41-9ad94cc2642c" path="/var/lib/kubelet/pods/c6a22f93-19ab-4222-9f41-9ad94cc2642c/volumes" Nov 03 02:19:50.645638 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:50.645588 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2" Nov 03 02:19:50.737860 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:50.737769 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fbbc88e-2111-44d1-a28d-537cc9aa1094-kube-api-access-6tfq7\" (UniqueName: \"kubernetes.io/projected/8fbbc88e-2111-44d1-a28d-537cc9aa1094-kube-api-access-6tfq7\") pod \"8fbbc88e-2111-44d1-a28d-537cc9aa1094\" (UID: \"8fbbc88e-2111-44d1-a28d-537cc9aa1094\") " Nov 03 02:19:50.738023 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:50.737931 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fbbc88e-2111-44d1-a28d-537cc9aa1094-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbbc88e-2111-44d1-a28d-537cc9aa1094-plugins-config-volume\") pod \"8fbbc88e-2111-44d1-a28d-537cc9aa1094\" (UID: \"8fbbc88e-2111-44d1-a28d-537cc9aa1094\") " Nov 03 02:19:50.738872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:50.738829 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fbbc88e-2111-44d1-a28d-537cc9aa1094-plugins-config-volume" pod "8fbbc88e-2111-44d1-a28d-537cc9aa1094" (UID: "8fbbc88e-2111-44d1-a28d-537cc9aa1094"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:19:50.743480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:50.743445 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbbc88e-2111-44d1-a28d-537cc9aa1094-kube-api-access-6tfq7" pod "8fbbc88e-2111-44d1-a28d-537cc9aa1094" (UID: "8fbbc88e-2111-44d1-a28d-537cc9aa1094"). InnerVolumeSpecName "kube-api-access-6tfq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:19:50.769051 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:50.769024 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9" Nov 03 02:19:50.839830 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:50.839801 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd144ab9-c755-4886-a81f-cbd747a5e5c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd144ab9-c755-4886-a81f-cbd747a5e5c3-plugins-config-volume\") pod \"cd144ab9-c755-4886-a81f-cbd747a5e5c3\" (UID: \"cd144ab9-c755-4886-a81f-cbd747a5e5c3\") " Nov 03 02:19:50.839991 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:50.839901 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd144ab9-c755-4886-a81f-cbd747a5e5c3-kube-api-access-zwqkx\" (UniqueName: \"kubernetes.io/projected/cd144ab9-c755-4886-a81f-cbd747a5e5c3-kube-api-access-zwqkx\") pod \"cd144ab9-c755-4886-a81f-cbd747a5e5c3\" (UID: \"cd144ab9-c755-4886-a81f-cbd747a5e5c3\") " Nov 03 02:19:50.840227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:50.840177 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd144ab9-c755-4886-a81f-cbd747a5e5c3-plugins-config-volume" pod "cd144ab9-c755-4886-a81f-cbd747a5e5c3" (UID: "cd144ab9-c755-4886-a81f-cbd747a5e5c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:19:50.840491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:50.840473 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd144ab9-c755-4886-a81f-cbd747a5e5c3-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:50.840491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:50.840490 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tfq7\" (UniqueName: \"kubernetes.io/projected/8fbbc88e-2111-44d1-a28d-537cc9aa1094-kube-api-access-6tfq7\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:50.840657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:50.840500 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbbc88e-2111-44d1-a28d-537cc9aa1094-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:50.842119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:50.842097 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd144ab9-c755-4886-a81f-cbd747a5e5c3-kube-api-access-zwqkx" pod "cd144ab9-c755-4886-a81f-cbd747a5e5c3" (UID: "cd144ab9-c755-4886-a81f-cbd747a5e5c3"). InnerVolumeSpecName "kube-api-access-zwqkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:19:50.941608 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:50.941568 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwqkx\" (UniqueName: \"kubernetes.io/projected/cd144ab9-c755-4886-a81f-cbd747a5e5c3-kube-api-access-zwqkx\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:51.329241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:51.329202 5463 generic.go:356] "Generic (PLEG): container finished" podID="8fbbc88e-2111-44d1-a28d-537cc9aa1094" containerID="eaf3617fcfdddabfa5740500b8149d8aaf0896cc9e34ff6395f79d061d46ef74" exitCode=0 Nov 03 02:19:51.329241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:51.329248 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2" Nov 03 02:19:51.329478 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:51.329283 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2" event={"ID":"8fbbc88e-2111-44d1-a28d-537cc9aa1094","Type":"ContainerDied","Data":"eaf3617fcfdddabfa5740500b8149d8aaf0896cc9e34ff6395f79d061d46ef74"} Nov 03 02:19:51.329478 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:51.329346 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2" event={"ID":"8fbbc88e-2111-44d1-a28d-537cc9aa1094","Type":"ContainerDied","Data":"8f8eaf26269eec8c7cabee2d929b80cb91b7a825698b61d30cd61870956ea8e3"} Nov 03 02:19:51.329478 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:51.329374 5463 scope.go:122] "RemoveContainer" containerID="eaf3617fcfdddabfa5740500b8149d8aaf0896cc9e34ff6395f79d061d46ef74" Nov 03 02:19:51.334818 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:51.334791 5463 generic.go:356] "Generic (PLEG): container finished" podID="cd144ab9-c755-4886-a81f-cbd747a5e5c3" containerID="7fb5872e8eaf0cf2079899b3d17f65f51a6061964249621a46e59128c8f1a234" exitCode=0 Nov 03 02:19:51.335105 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:51.334848 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9" Nov 03 02:19:51.335425 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:51.334865 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9" event={"ID":"cd144ab9-c755-4886-a81f-cbd747a5e5c3","Type":"ContainerDied","Data":"7fb5872e8eaf0cf2079899b3d17f65f51a6061964249621a46e59128c8f1a234"} Nov 03 02:19:51.335575 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:51.335558 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9" event={"ID":"cd144ab9-c755-4886-a81f-cbd747a5e5c3","Type":"ContainerDied","Data":"ab06aaaa149f0a050dfe819a32a5d51cf541cbc099e10d9cccb349ef9cf2ccc4"} Nov 03 02:19:51.355359 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:51.355282 5463 scope.go:122] "RemoveContainer" containerID="eaf3617fcfdddabfa5740500b8149d8aaf0896cc9e34ff6395f79d061d46ef74" Nov 03 02:19:51.355721 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:19:51.355691 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaf3617fcfdddabfa5740500b8149d8aaf0896cc9e34ff6395f79d061d46ef74\": container with ID starting with eaf3617fcfdddabfa5740500b8149d8aaf0896cc9e34ff6395f79d061d46ef74 not found: ID does not exist" containerID="eaf3617fcfdddabfa5740500b8149d8aaf0896cc9e34ff6395f79d061d46ef74" Nov 03 02:19:51.355817 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:51.355731 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaf3617fcfdddabfa5740500b8149d8aaf0896cc9e34ff6395f79d061d46ef74"} err="failed to get container status \"eaf3617fcfdddabfa5740500b8149d8aaf0896cc9e34ff6395f79d061d46ef74\": rpc error: code = NotFound desc = could not find container \"eaf3617fcfdddabfa5740500b8149d8aaf0896cc9e34ff6395f79d061d46ef74\": container with ID starting with eaf3617fcfdddabfa5740500b8149d8aaf0896cc9e34ff6395f79d061d46ef74 not found: ID does not exist" Nov 03 02:19:51.355817 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:51.355755 5463 scope.go:122] "RemoveContainer" containerID="7fb5872e8eaf0cf2079899b3d17f65f51a6061964249621a46e59128c8f1a234" Nov 03 02:19:51.380178 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:51.379709 5463 scope.go:122] "RemoveContainer" containerID="7fb5872e8eaf0cf2079899b3d17f65f51a6061964249621a46e59128c8f1a234" Nov 03 02:19:51.380178 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:19:51.380077 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb5872e8eaf0cf2079899b3d17f65f51a6061964249621a46e59128c8f1a234\": container with ID starting with 7fb5872e8eaf0cf2079899b3d17f65f51a6061964249621a46e59128c8f1a234 not found: ID does not exist" containerID="7fb5872e8eaf0cf2079899b3d17f65f51a6061964249621a46e59128c8f1a234" Nov 03 02:19:51.380178 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:51.380112 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb5872e8eaf0cf2079899b3d17f65f51a6061964249621a46e59128c8f1a234"} err="failed to get container status \"7fb5872e8eaf0cf2079899b3d17f65f51a6061964249621a46e59128c8f1a234\": rpc error: code = NotFound desc = could not find container \"7fb5872e8eaf0cf2079899b3d17f65f51a6061964249621a46e59128c8f1a234\": container with ID starting with 7fb5872e8eaf0cf2079899b3d17f65f51a6061964249621a46e59128c8f1a234 not found: ID does not exist" Nov 03 02:19:51.495828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:51.495786 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9"] Nov 03 02:19:51.546435 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:51.546393 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-596d7d688c-9mvd9"] Nov 03 02:19:51.612076 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:51.611988 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2"] Nov 03 02:19:51.617152 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:51.617120 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-84559dd8d7-ps4g2"] Nov 03 02:19:52.111430 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:52.111394 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:52.347523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:52.347445 5463 generic.go:356] "Generic (PLEG): container finished" podID="a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" containerID="99dca0b9f302869d45fae0b9555ae0f00430ab14f5c2a3ff1b6d01822a39155a" exitCode=0 Nov 03 02:19:52.347697 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:52.347524 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" event={"ID":"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd","Type":"ContainerDied","Data":"99dca0b9f302869d45fae0b9555ae0f00430ab14f5c2a3ff1b6d01822a39155a"} Nov 03 02:19:52.348006 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:52.347986 5463 scope.go:122] "RemoveContainer" containerID="99dca0b9f302869d45fae0b9555ae0f00430ab14f5c2a3ff1b6d01822a39155a" Nov 03 02:19:52.349722 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:52.349691 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:19:52.349804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:52.349769 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:19:52.349841 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:52.349807 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:19:52.425181 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:52.425148 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:19:52.446018 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:52.445727 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fbbc88e-2111-44d1-a28d-537cc9aa1094" path="/var/lib/kubelet/pods/8fbbc88e-2111-44d1-a28d-537cc9aa1094/volumes" Nov 03 02:19:52.446356 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:52.446337 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd144ab9-c755-4886-a81f-cbd747a5e5c3" path="/var/lib/kubelet/pods/cd144ab9-c755-4886-a81f-cbd747a5e5c3/volumes" Nov 03 02:19:53.382045 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.382005 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" event={"ID":"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd","Type":"ContainerStarted","Data":"1581ecb527924e3d11fd74ba28f7b3ebced6afefd88445db0a33541ae86abadd"} Nov 03 02:19:53.496838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.496805 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:53.647093 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.647054 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:53.857216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.847939 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:53.878212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.876536 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:19:53.883213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.881481 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 02:19:53.883213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.881552 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" podUID="a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 02:19:53.895589 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.895559 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg"] Nov 03 02:19:53.896185 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.896172 5463 cpu_manager.go:414] "containerMap: removing container" podUID="bd79820a-5bdb-449a-9e7f-559755e6e9ee" containerName="payload-processing" Nov 03 02:19:53.896275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.896186 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd79820a-5bdb-449a-9e7f-559755e6e9ee" containerName="payload-processing" Nov 03 02:19:53.896275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.896215 5463 cpu_manager.go:414] "containerMap: removing container" podUID="cd144ab9-c755-4886-a81f-cbd747a5e5c3" containerName="payload-pre-processing" Nov 03 02:19:53.896275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.896220 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd144ab9-c755-4886-a81f-cbd747a5e5c3" containerName="payload-pre-processing" Nov 03 02:19:53.896275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.896252 5463 cpu_manager.go:414] "containerMap: removing container" podUID="8fbbc88e-2111-44d1-a28d-537cc9aa1094" containerName="payload-processing" Nov 03 02:19:53.896275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.896257 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fbbc88e-2111-44d1-a28d-537cc9aa1094" containerName="payload-processing" Nov 03 02:19:53.896275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.896270 5463 cpu_manager.go:414] "containerMap: removing container" podUID="c6a22f93-19ab-4222-9f41-9ad94cc2642c" containerName="payload-pre-processing" Nov 03 02:19:53.896275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.896275 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6a22f93-19ab-4222-9f41-9ad94cc2642c" containerName="payload-pre-processing" Nov 03 02:19:53.896499 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.896368 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fbbc88e-2111-44d1-a28d-537cc9aa1094" containerName="payload-processing" Nov 03 02:19:53.896499 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.896379 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6a22f93-19ab-4222-9f41-9ad94cc2642c" containerName="payload-pre-processing" Nov 03 02:19:53.896499 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.896388 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd79820a-5bdb-449a-9e7f-559755e6e9ee" containerName="payload-processing" Nov 03 02:19:53.896499 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.896395 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd144ab9-c755-4886-a81f-cbd747a5e5c3" containerName="payload-pre-processing" Nov 03 02:19:53.899526 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.899477 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg" Nov 03 02:19:53.979348 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.979086 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5ef2c8\"" Nov 03 02:19:53.989247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.986391 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5ef2c8-dockercfg-ldqqs\"" Nov 03 02:19:53.999332 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:53.999299 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg"] Nov 03 02:19:54.020315 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.007054 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c007625-c8c9-4e25-980c-23d59b9725d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg\" (UID: \"4c007625-c8c9-4e25-980c-23d59b9725d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg" Nov 03 02:19:54.020315 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.007116 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfhsn\" (UniqueName: \"kubernetes.io/projected/4c007625-c8c9-4e25-980c-23d59b9725d6-kube-api-access-kfhsn\") pod \"payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg\" (UID: \"4c007625-c8c9-4e25-980c-23d59b9725d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg" Nov 03 02:19:54.086218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.048456 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv"] Nov 03 02:19:54.097469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.096481 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv" Nov 03 02:19:54.101346 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.100839 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv"] Nov 03 02:19:54.114774 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.109066 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c007625-c8c9-4e25-980c-23d59b9725d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg\" (UID: \"4c007625-c8c9-4e25-980c-23d59b9725d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg" Nov 03 02:19:54.114774 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.109631 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfhsn\" (UniqueName: \"kubernetes.io/projected/4c007625-c8c9-4e25-980c-23d59b9725d6-kube-api-access-kfhsn\") pod \"payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg\" (UID: \"4c007625-c8c9-4e25-980c-23d59b9725d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg" Nov 03 02:19:54.114774 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.110893 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c007625-c8c9-4e25-980c-23d59b9725d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg\" (UID: \"4c007625-c8c9-4e25-980c-23d59b9725d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg" Nov 03 02:19:54.185828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.185032 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfhsn\" (UniqueName: \"kubernetes.io/projected/4c007625-c8c9-4e25-980c-23d59b9725d6-kube-api-access-kfhsn\") pod \"payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg\" (UID: \"4c007625-c8c9-4e25-980c-23d59b9725d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg" Nov 03 02:19:54.212153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.212125 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5549425a-e04c-4436-bf47-def96605061e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv\" (UID: \"5549425a-e04c-4436-bf47-def96605061e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv" Nov 03 02:19:54.212348 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.212326 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq4n2\" (UniqueName: \"kubernetes.io/projected/5549425a-e04c-4436-bf47-def96605061e-kube-api-access-nq4n2\") pod \"payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv\" (UID: \"5549425a-e04c-4436-bf47-def96605061e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv" Nov 03 02:19:54.212558 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.212543 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg" Nov 03 02:19:54.314810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.314778 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5549425a-e04c-4436-bf47-def96605061e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv\" (UID: \"5549425a-e04c-4436-bf47-def96605061e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv" Nov 03 02:19:54.314999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.314981 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq4n2\" (UniqueName: \"kubernetes.io/projected/5549425a-e04c-4436-bf47-def96605061e-kube-api-access-nq4n2\") pod \"payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv\" (UID: \"5549425a-e04c-4436-bf47-def96605061e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv" Nov 03 02:19:54.316240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.316216 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5549425a-e04c-4436-bf47-def96605061e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv\" (UID: \"5549425a-e04c-4436-bf47-def96605061e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv" Nov 03 02:19:54.382982 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.382942 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq4n2\" (UniqueName: \"kubernetes.io/projected/5549425a-e04c-4436-bf47-def96605061e-kube-api-access-nq4n2\") pod \"payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv\" (UID: \"5549425a-e04c-4436-bf47-def96605061e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv" Nov 03 02:19:54.421448 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.421419 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv" Nov 03 02:19:54.605063 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:19:54.605026 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c007625_c8c9_4e25_980c_23d59b9725d6.slice/crio-72026cdb1872853f519c3cf68707c6d79a276ff139a862f1731fe0ec1e72d3b9 WatchSource:0}: Error finding container 72026cdb1872853f519c3cf68707c6d79a276ff139a862f1731fe0ec1e72d3b9: Status 404 returned error can't find the container with id 72026cdb1872853f519c3cf68707c6d79a276ff139a862f1731fe0ec1e72d3b9 Nov 03 02:19:54.608351 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.608330 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg"] Nov 03 02:19:54.816542 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:19:54.816491 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5549425a_e04c_4436_bf47_def96605061e.slice/crio-3a9890df5922d9246be400883008b0dfd9b472048ac245fcc4e6b65e1357db54 WatchSource:0}: Error finding container 3a9890df5922d9246be400883008b0dfd9b472048ac245fcc4e6b65e1357db54: Status 404 returned error can't find the container with id 3a9890df5922d9246be400883008b0dfd9b472048ac245fcc4e6b65e1357db54 Nov 03 02:19:54.820354 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.820328 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv"] Nov 03 02:19:54.871534 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.871425 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 02:19:54.871534 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:54.871479 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" podUID="a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 02:19:55.014432 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:55.014399 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:55.298832 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:55.298795 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:55.398906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:55.398871 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg" event={"ID":"4c007625-c8c9-4e25-980c-23d59b9725d6","Type":"ContainerStarted","Data":"0f29cab8f980fa34bd5a3bdb79204b07c444848fca9daf3ee37d82a527a6ba01"} Nov 03 02:19:55.399380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:55.398939 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg" Nov 03 02:19:55.399380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:55.398967 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg" event={"ID":"4c007625-c8c9-4e25-980c-23d59b9725d6","Type":"ContainerStarted","Data":"72026cdb1872853f519c3cf68707c6d79a276ff139a862f1731fe0ec1e72d3b9"} Nov 03 02:19:55.400653 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:55.400629 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv" event={"ID":"5549425a-e04c-4436-bf47-def96605061e","Type":"ContainerStarted","Data":"6c3f2e0a6f7ea557b6520ce98410c37f2affaa97e6d1ebe9f8693aa37cec6cd0"} Nov 03 02:19:55.400787 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:55.400677 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv" event={"ID":"5549425a-e04c-4436-bf47-def96605061e","Type":"ContainerStarted","Data":"3a9890df5922d9246be400883008b0dfd9b472048ac245fcc4e6b65e1357db54"} Nov 03 02:19:55.400858 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:55.400792 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv" Nov 03 02:19:55.504938 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:55.504894 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:55.522810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:55.522750 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg" podStartSLOduration=2.522729303 podStartE2EDuration="2.522729303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:19:55.518859247 +0000 UTC m=+1677.202968400" watchObservedRunningTime="2026-11-03 02:19:55.522729303 +0000 UTC m=+1677.206838459" Nov 03 02:19:55.635073 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:55.634979 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv" podStartSLOduration=2.634963356 podStartE2EDuration="2.634963356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:19:55.633831348 +0000 UTC m=+1677.317940504" watchObservedRunningTime="2026-11-03 02:19:55.634963356 +0000 UTC m=+1677.319072542" Nov 03 02:19:55.691221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:55.659613 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:55.871577 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:55.871538 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 02:19:55.871846 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:55.871817 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" podUID="a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 02:19:56.425010 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:56.424966 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:56.822877 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:56.822789 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:56.878773 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:56.878738 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 02:19:56.878958 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:56.878798 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" podUID="a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 02:19:57.274214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.273966 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz"] Nov 03 02:19:57.278310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.278282 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.332183 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.332151 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9"] Nov 03 02:19:57.332420 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.332394 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" podUID="2fd41e7e-553d-45c9-b25c-d4aa074c899d" containerName="istio-proxy" containerID="cri-o://db22fc899b5d467fc88580f4fa82cc84e10d1b608b5602a0c266894e8399eedf" gracePeriod=30 Nov 03 02:19:57.334849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.334804 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:57.335089 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.335066 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3bbfc4-openshift-default-dockercfg-bljtg\"" Nov 03 02:19:57.390040 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.389989 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz"] Nov 03 02:19:57.392800 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.392755 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-workload-certs\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.394739 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.394699 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-data\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.397275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.397244 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-envoy\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.397458 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.397431 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-credential-socket\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.397613 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.397587 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-token\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.397761 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.397737 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57gq6\" (UniqueName: \"kubernetes.io/projected/871baea4-93af-43d2-8ef7-0b05c4f50526-kube-api-access-57gq6\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.397943 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.397919 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871baea4-93af-43d2-8ef7-0b05c4f50526-istiod-ca-cert\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.398131 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.398109 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-workload-socket\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.398305 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.398280 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-podinfo\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.511214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.505503 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-podinfo\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.511214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.505695 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-workload-certs\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.511214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.505827 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-data\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.511214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.506076 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-envoy\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.523214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.514678 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-data\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.523214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.515442 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-credential-socket\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.523214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.515573 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-token\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.523214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.515606 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57gq6\" (UniqueName: \"kubernetes.io/projected/871baea4-93af-43d2-8ef7-0b05c4f50526-kube-api-access-57gq6\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.523214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.515760 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871baea4-93af-43d2-8ef7-0b05c4f50526-istiod-ca-cert\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.523214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.515798 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-credential-socket\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.523214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.515897 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-workload-socket\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.523214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.516004 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-workload-certs\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.523214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.516327 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-workload-socket\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.523214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.516855 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871baea4-93af-43d2-8ef7-0b05c4f50526-istiod-ca-cert\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.523214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.521804 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:57.536286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.535551 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-envoy\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.540765 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.540732 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-podinfo\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.577213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.574878 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57gq6\" (UniqueName: \"kubernetes.io/projected/871baea4-93af-43d2-8ef7-0b05c4f50526-kube-api-access-57gq6\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.600250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.599788 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-token\") pod \"e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.600250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.600039 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:57.685886 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.685133 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:57.873265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.872597 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 03 02:19:57.873265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:57.872681 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" podUID="a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 03 02:19:58.080225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:58.071937 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz"] Nov 03 02:19:58.080225 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:19:58.074296 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod871baea4_93af_43d2_8ef7_0b05c4f50526.slice/crio-6cd63137b1832d91c14adfcdf6d52898179d225d3b91ed1b5bf54e274c094401 WatchSource:0}: Error finding container 6cd63137b1832d91c14adfcdf6d52898179d225d3b91ed1b5bf54e274c094401: Status 404 returned error can't find the container with id 6cd63137b1832d91c14adfcdf6d52898179d225d3b91ed1b5bf54e274c094401 Nov 03 02:19:58.084934 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:58.084470 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:19:58.084934 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:58.084559 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:19:58.084934 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:58.084593 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:19:58.168879 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:58.168808 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:58.535556 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:58.535514 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" event={"ID":"871baea4-93af-43d2-8ef7-0b05c4f50526","Type":"ContainerStarted","Data":"58896957410a6ed50693266f6766b0bf5a3019aa55b56c1fbc1a520a40c4effb"} Nov 03 02:19:58.536105 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:58.536074 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" event={"ID":"871baea4-93af-43d2-8ef7-0b05c4f50526","Type":"ContainerStarted","Data":"6cd63137b1832d91c14adfcdf6d52898179d225d3b91ed1b5bf54e274c094401"} Nov 03 02:19:58.601076 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:58.601046 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:19:58.653769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:58.653723 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" podStartSLOduration=1.6537085679999999 podStartE2EDuration="1.653708568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:19:58.651317446 +0000 UTC m=+1680.335426599" watchObservedRunningTime="2026-11-03 02:19:58.653708568 +0000 UTC m=+1680.337817722" Nov 03 02:19:58.878000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:58.877919 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:19:58.878447 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:58.878422 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:19:58.879297 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:58.879280 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:19:59.606511 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:19:59.606482 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:20:00.549549 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:00.549512 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:20:00.550629 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:00.550602 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:20:01.446684 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:01.446657 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:01.624082 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:01.624046 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:01.781431 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:01.781355 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:02.570349 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.570316 5463 generic.go:356] "Generic (PLEG): container finished" podID="2fd41e7e-553d-45c9-b25c-d4aa074c899d" containerID="db22fc899b5d467fc88580f4fa82cc84e10d1b608b5602a0c266894e8399eedf" exitCode=0 Nov 03 02:20:02.570813 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.570396 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" event={"ID":"2fd41e7e-553d-45c9-b25c-d4aa074c899d","Type":"ContainerDied","Data":"db22fc899b5d467fc88580f4fa82cc84e10d1b608b5602a0c266894e8399eedf"} Nov 03 02:20:02.570813 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.570458 5463 scope.go:122] "RemoveContainer" containerID="537e46e20db1abb465a30d70c651246ee409315f3744de348bd838ffee9dfb5a" Nov 03 02:20:02.611964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.611915 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:20:02.623849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.623828 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:02.681761 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.681732 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-workload-certs\") pod \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " Nov 03 02:20:02.681921 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.681774 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd41e7e-553d-45c9-b25c-d4aa074c899d-kube-api-access-gvm25\" (UniqueName: \"kubernetes.io/projected/2fd41e7e-553d-45c9-b25c-d4aa074c899d-kube-api-access-gvm25\") pod \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " Nov 03 02:20:02.681921 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.681829 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-data\") pod \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " Nov 03 02:20:02.681921 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.681847 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-envoy\") pod \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " Nov 03 02:20:02.682050 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.681968 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-podinfo\") pod \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " Nov 03 02:20:02.682050 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.682009 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istiod-ca-cert\") pod \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " Nov 03 02:20:02.682234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.682049 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-workload-certs" pod "2fd41e7e-553d-45c9-b25c-d4aa074c899d" (UID: "2fd41e7e-553d-45c9-b25c-d4aa074c899d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:02.682234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.682088 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-credential-socket\") pod \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " Nov 03 02:20:02.682234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.682160 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-token\" (UniqueName: \"kubernetes.io/projected/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-token\") pod \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " Nov 03 02:20:02.682234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.682220 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-workload-socket\") pod \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\" (UID: \"2fd41e7e-553d-45c9-b25c-d4aa074c899d\") " Nov 03 02:20:02.682459 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.682303 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-data" pod "2fd41e7e-553d-45c9-b25c-d4aa074c899d" (UID: "2fd41e7e-553d-45c9-b25c-d4aa074c899d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:02.682459 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.682360 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-credential-socket" pod "2fd41e7e-553d-45c9-b25c-d4aa074c899d" (UID: "2fd41e7e-553d-45c9-b25c-d4aa074c899d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:02.682459 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.682412 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istiod-ca-cert" pod "2fd41e7e-553d-45c9-b25c-d4aa074c899d" (UID: "2fd41e7e-553d-45c9-b25c-d4aa074c899d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:20:02.683307 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.682633 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-workload-socket" pod "2fd41e7e-553d-45c9-b25c-d4aa074c899d" (UID: "2fd41e7e-553d-45c9-b25c-d4aa074c899d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:02.683307 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.682972 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:02.683307 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.682993 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:02.683307 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.683008 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:02.683307 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.683022 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:02.683307 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.683038 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:02.684510 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.684488 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-token" pod "2fd41e7e-553d-45c9-b25c-d4aa074c899d" (UID: "2fd41e7e-553d-45c9-b25c-d4aa074c899d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:20:02.684646 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.684552 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd41e7e-553d-45c9-b25c-d4aa074c899d-kube-api-access-gvm25" pod "2fd41e7e-553d-45c9-b25c-d4aa074c899d" (UID: "2fd41e7e-553d-45c9-b25c-d4aa074c899d"). InnerVolumeSpecName "kube-api-access-gvm25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:20:02.684774 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.684748 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-envoy" pod "2fd41e7e-553d-45c9-b25c-d4aa074c899d" (UID: "2fd41e7e-553d-45c9-b25c-d4aa074c899d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:02.684836 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.684817 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-podinfo" pod "2fd41e7e-553d-45c9-b25c-d4aa074c899d" (UID: "2fd41e7e-553d-45c9-b25c-d4aa074c899d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:20:02.784515 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.784486 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:02.784515 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.784511 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvm25\" (UniqueName: \"kubernetes.io/projected/2fd41e7e-553d-45c9-b25c-d4aa074c899d-kube-api-access-gvm25\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:02.784515 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.784521 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:02.784729 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:02.784530 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd41e7e-553d-45c9-b25c-d4aa074c899d-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:03.579580 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:03.579543 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" Nov 03 02:20:03.580103 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:03.579585 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9" event={"ID":"2fd41e7e-553d-45c9-b25c-d4aa074c899d","Type":"ContainerDied","Data":"8780147d68f9afe90f91ee1421b64fc56f0491b9d60857bc6b139918110600ae"} Nov 03 02:20:03.580103 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:03.579645 5463 scope.go:122] "RemoveContainer" containerID="db22fc899b5d467fc88580f4fa82cc84e10d1b608b5602a0c266894e8399eedf" Nov 03 02:20:03.796105 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:03.796069 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9"] Nov 03 02:20:03.846246 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:03.846151 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6198c6-openshift-default-fd4cf889f-s5ch9"] Nov 03 02:20:04.313677 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:04.313644 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:04.443534 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:04.443493 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fd41e7e-553d-45c9-b25c-d4aa074c899d" path="/var/lib/kubelet/pods/2fd41e7e-553d-45c9-b25c-d4aa074c899d/volumes" Nov 03 02:20:04.689650 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:04.689616 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:04.865693 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:04.865614 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:05.067136 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.067103 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw"] Nov 03 02:20:05.067751 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.067736 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2fd41e7e-553d-45c9-b25c-d4aa074c899d" containerName="istio-proxy" Nov 03 02:20:05.067800 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.067753 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd41e7e-553d-45c9-b25c-d4aa074c899d" containerName="istio-proxy" Nov 03 02:20:05.067800 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.067787 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2fd41e7e-553d-45c9-b25c-d4aa074c899d" containerName="istio-proxy" Nov 03 02:20:05.067800 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.067792 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd41e7e-553d-45c9-b25c-d4aa074c899d" containerName="istio-proxy" Nov 03 02:20:05.067918 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.067901 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd41e7e-553d-45c9-b25c-d4aa074c899d" containerName="istio-proxy" Nov 03 02:20:05.072052 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.072032 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw" Nov 03 02:20:05.124080 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.124015 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw"] Nov 03 02:20:05.124541 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.124517 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3bbfc4\"" Nov 03 02:20:05.124678 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.124649 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3bbfc4-dockercfg-zlt7s\"" Nov 03 02:20:05.124839 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.124817 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c39cbfb-d4b9-49c9-96be-e93ae28e9d62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw\" (UID: \"6c39cbfb-d4b9-49c9-96be-e93ae28e9d62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw" Nov 03 02:20:05.125034 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.125018 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ph6f\" (UniqueName: \"kubernetes.io/projected/6c39cbfb-d4b9-49c9-96be-e93ae28e9d62-kube-api-access-4ph6f\") pod \"payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw\" (UID: \"6c39cbfb-d4b9-49c9-96be-e93ae28e9d62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw" Nov 03 02:20:05.227614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.226306 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ph6f\" (UniqueName: \"kubernetes.io/projected/6c39cbfb-d4b9-49c9-96be-e93ae28e9d62-kube-api-access-4ph6f\") pod \"payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw\" (UID: \"6c39cbfb-d4b9-49c9-96be-e93ae28e9d62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw" Nov 03 02:20:05.227614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.226485 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c39cbfb-d4b9-49c9-96be-e93ae28e9d62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw\" (UID: \"6c39cbfb-d4b9-49c9-96be-e93ae28e9d62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw" Nov 03 02:20:05.227614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.227561 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c39cbfb-d4b9-49c9-96be-e93ae28e9d62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw\" (UID: \"6c39cbfb-d4b9-49c9-96be-e93ae28e9d62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw" Nov 03 02:20:05.239761 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.239731 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn"] Nov 03 02:20:05.242349 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.240550 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd41e7e-553d-45c9-b25c-d4aa074c899d" containerName="istio-proxy" Nov 03 02:20:05.243227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.243173 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn" Nov 03 02:20:05.283467 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.283436 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ph6f\" (UniqueName: \"kubernetes.io/projected/6c39cbfb-d4b9-49c9-96be-e93ae28e9d62-kube-api-access-4ph6f\") pod \"payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw\" (UID: \"6c39cbfb-d4b9-49c9-96be-e93ae28e9d62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw" Nov 03 02:20:05.299678 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.298644 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn"] Nov 03 02:20:05.327943 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.327912 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e7c13a-9588-4e93-b247-1830f7ba06b7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn\" (UID: \"69e7c13a-9588-4e93-b247-1830f7ba06b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn" Nov 03 02:20:05.328119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.328099 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdwp7\" (UniqueName: \"kubernetes.io/projected/69e7c13a-9588-4e93-b247-1830f7ba06b7-kube-api-access-kdwp7\") pod \"payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn\" (UID: \"69e7c13a-9588-4e93-b247-1830f7ba06b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn" Nov 03 02:20:05.385821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.385756 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw" Nov 03 02:20:05.429527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.429496 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e7c13a-9588-4e93-b247-1830f7ba06b7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn\" (UID: \"69e7c13a-9588-4e93-b247-1830f7ba06b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn" Nov 03 02:20:05.429691 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.429663 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdwp7\" (UniqueName: \"kubernetes.io/projected/69e7c13a-9588-4e93-b247-1830f7ba06b7-kube-api-access-kdwp7\") pod \"payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn\" (UID: \"69e7c13a-9588-4e93-b247-1830f7ba06b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn" Nov 03 02:20:05.430624 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.430597 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e7c13a-9588-4e93-b247-1830f7ba06b7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn\" (UID: \"69e7c13a-9588-4e93-b247-1830f7ba06b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn" Nov 03 02:20:05.492247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.491019 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdwp7\" (UniqueName: \"kubernetes.io/projected/69e7c13a-9588-4e93-b247-1830f7ba06b7-kube-api-access-kdwp7\") pod \"payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn\" (UID: \"69e7c13a-9588-4e93-b247-1830f7ba06b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn" Nov 03 02:20:05.562004 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.561962 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn" Nov 03 02:20:05.755587 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:20:05.755557 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c39cbfb_d4b9_49c9_96be_e93ae28e9d62.slice/crio-26288d24c3fd7e7b940dd3c433ca63f5bdd7cded7fd4e469a53ccf65aa29b235 WatchSource:0}: Error finding container 26288d24c3fd7e7b940dd3c433ca63f5bdd7cded7fd4e469a53ccf65aa29b235: Status 404 returned error can't find the container with id 26288d24c3fd7e7b940dd3c433ca63f5bdd7cded7fd4e469a53ccf65aa29b235 Nov 03 02:20:05.758971 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.758892 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw"] Nov 03 02:20:05.800287 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.800242 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x"] Nov 03 02:20:05.904109 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.904082 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf"] Nov 03 02:20:05.950285 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:20:05.948569 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e7c13a_9588_4e93_b247_1830f7ba06b7.slice/crio-0a3eb1fbcaa68d37620fe0c521f0501852399d2e806bbca47d68372be737490b WatchSource:0}: Error finding container 0a3eb1fbcaa68d37620fe0c521f0501852399d2e806bbca47d68372be737490b: Status 404 returned error can't find the container with id 0a3eb1fbcaa68d37620fe0c521f0501852399d2e806bbca47d68372be737490b Nov 03 02:20:05.952069 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:05.952046 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn"] Nov 03 02:20:06.174361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:06.174329 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:06.330478 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:06.330400 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:06.413239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:06.413177 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg" Nov 03 02:20:06.413509 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:06.413489 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv" Nov 03 02:20:06.601247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:06.601125 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw" event={"ID":"6c39cbfb-d4b9-49c9-96be-e93ae28e9d62","Type":"ContainerStarted","Data":"e22725dea4a2d24c482f423d813f090f9e295d1f34044ff2131def4bdc126465"} Nov 03 02:20:06.601247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:06.601185 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw" event={"ID":"6c39cbfb-d4b9-49c9-96be-e93ae28e9d62","Type":"ContainerStarted","Data":"26288d24c3fd7e7b940dd3c433ca63f5bdd7cded7fd4e469a53ccf65aa29b235"} Nov 03 02:20:06.601247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:06.601223 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw" Nov 03 02:20:06.602783 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:06.602761 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn" event={"ID":"69e7c13a-9588-4e93-b247-1830f7ba06b7","Type":"ContainerStarted","Data":"469c94a12bfc770ec9e130f8a708d2f7576418e61ead9e4780b8251f66d5bf54"} Nov 03 02:20:06.602876 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:06.602797 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn" event={"ID":"69e7c13a-9588-4e93-b247-1830f7ba06b7","Type":"ContainerStarted","Data":"0a3eb1fbcaa68d37620fe0c521f0501852399d2e806bbca47d68372be737490b"} Nov 03 02:20:06.602876 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:06.602845 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn" Nov 03 02:20:06.757916 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:06.757848 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw" podStartSLOduration=1.75783033 podStartE2EDuration="1.75783033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:20:06.752343214 +0000 UTC m=+1688.436452369" watchObservedRunningTime="2026-11-03 02:20:06.75783033 +0000 UTC m=+1688.441939486" Nov 03 02:20:06.876982 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:06.876859 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn" podStartSLOduration=1.8768438600000001 podStartE2EDuration="1.87684386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:20:06.875585187 +0000 UTC m=+1688.559694343" watchObservedRunningTime="2026-11-03 02:20:06.87684386 +0000 UTC m=+1688.560953014" Nov 03 02:20:07.825319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:07.825269 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:09.957068 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:09.957000 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:10.800725 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:10.800665 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x" podUID="db83203f-b4b2-467f-9fb7-b105d5e89751" containerName="payload-processing" containerID="cri-o://490121feb5482e557715f4df6bcd4c78f83135439d51a419166e1490406fd625" gracePeriod=25 Nov 03 02:20:10.904873 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:10.904833 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf" podUID="82305fbb-af59-4971-bb94-137cff38d00b" containerName="payload-pre-processing" containerID="cri-o://b9bacf70db550e4eb4dbd260269a15b28baab037ca29e0d3fb0f311a4313e9bd" gracePeriod=25 Nov 03 02:20:11.100564 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.100533 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x" Nov 03 02:20:11.207128 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.207104 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf" Nov 03 02:20:11.214539 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.214504 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db83203f-b4b2-467f-9fb7-b105d5e89751-kube-api-access-zszr6\" (UniqueName: \"kubernetes.io/projected/db83203f-b4b2-467f-9fb7-b105d5e89751-kube-api-access-zszr6\") pod \"db83203f-b4b2-467f-9fb7-b105d5e89751\" (UID: \"db83203f-b4b2-467f-9fb7-b105d5e89751\") " Nov 03 02:20:11.214675 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.214632 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db83203f-b4b2-467f-9fb7-b105d5e89751-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db83203f-b4b2-467f-9fb7-b105d5e89751-plugins-config-volume\") pod \"db83203f-b4b2-467f-9fb7-b105d5e89751\" (UID: \"db83203f-b4b2-467f-9fb7-b105d5e89751\") " Nov 03 02:20:11.215071 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.215035 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db83203f-b4b2-467f-9fb7-b105d5e89751-plugins-config-volume" pod "db83203f-b4b2-467f-9fb7-b105d5e89751" (UID: "db83203f-b4b2-467f-9fb7-b105d5e89751"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:20:11.215386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.215360 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db83203f-b4b2-467f-9fb7-b105d5e89751-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:11.217096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.217068 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db83203f-b4b2-467f-9fb7-b105d5e89751-kube-api-access-zszr6" pod "db83203f-b4b2-467f-9fb7-b105d5e89751" (UID: "db83203f-b4b2-467f-9fb7-b105d5e89751"). InnerVolumeSpecName "kube-api-access-zszr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:20:11.316330 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.316253 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82305fbb-af59-4971-bb94-137cff38d00b-kube-api-access-vf6zw\" (UniqueName: \"kubernetes.io/projected/82305fbb-af59-4971-bb94-137cff38d00b-kube-api-access-vf6zw\") pod \"82305fbb-af59-4971-bb94-137cff38d00b\" (UID: \"82305fbb-af59-4971-bb94-137cff38d00b\") " Nov 03 02:20:11.316484 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.316356 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82305fbb-af59-4971-bb94-137cff38d00b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82305fbb-af59-4971-bb94-137cff38d00b-plugins-config-volume\") pod \"82305fbb-af59-4971-bb94-137cff38d00b\" (UID: \"82305fbb-af59-4971-bb94-137cff38d00b\") " Nov 03 02:20:11.316744 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.316725 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82305fbb-af59-4971-bb94-137cff38d00b-plugins-config-volume" pod "82305fbb-af59-4971-bb94-137cff38d00b" (UID: "82305fbb-af59-4971-bb94-137cff38d00b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:20:11.316792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.316743 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zszr6\" (UniqueName: \"kubernetes.io/projected/db83203f-b4b2-467f-9fb7-b105d5e89751-kube-api-access-zszr6\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:11.318439 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.318420 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82305fbb-af59-4971-bb94-137cff38d00b-kube-api-access-vf6zw" pod "82305fbb-af59-4971-bb94-137cff38d00b" (UID: "82305fbb-af59-4971-bb94-137cff38d00b"). InnerVolumeSpecName "kube-api-access-vf6zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:20:11.417915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.417882 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf6zw\" (UniqueName: \"kubernetes.io/projected/82305fbb-af59-4971-bb94-137cff38d00b-kube-api-access-vf6zw\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:11.417915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.417910 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82305fbb-af59-4971-bb94-137cff38d00b-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:11.637335 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.637256 5463 generic.go:356] "Generic (PLEG): container finished" podID="db83203f-b4b2-467f-9fb7-b105d5e89751" containerID="490121feb5482e557715f4df6bcd4c78f83135439d51a419166e1490406fd625" exitCode=0 Nov 03 02:20:11.637335 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.637302 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x" Nov 03 02:20:11.637537 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.637336 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x" event={"ID":"db83203f-b4b2-467f-9fb7-b105d5e89751","Type":"ContainerDied","Data":"490121feb5482e557715f4df6bcd4c78f83135439d51a419166e1490406fd625"} Nov 03 02:20:11.637537 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.637387 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x" event={"ID":"db83203f-b4b2-467f-9fb7-b105d5e89751","Type":"ContainerDied","Data":"3c676124f3298c3452bec6f62a287861ba66645e3588950ac02f8355e16403ac"} Nov 03 02:20:11.637537 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.637412 5463 scope.go:122] "RemoveContainer" containerID="490121feb5482e557715f4df6bcd4c78f83135439d51a419166e1490406fd625" Nov 03 02:20:11.638792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.638768 5463 generic.go:356] "Generic (PLEG): container finished" podID="82305fbb-af59-4971-bb94-137cff38d00b" containerID="b9bacf70db550e4eb4dbd260269a15b28baab037ca29e0d3fb0f311a4313e9bd" exitCode=0 Nov 03 02:20:11.638921 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.638835 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf" Nov 03 02:20:11.638921 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.638853 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf" event={"ID":"82305fbb-af59-4971-bb94-137cff38d00b","Type":"ContainerDied","Data":"b9bacf70db550e4eb4dbd260269a15b28baab037ca29e0d3fb0f311a4313e9bd"} Nov 03 02:20:11.639014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.638928 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf" event={"ID":"82305fbb-af59-4971-bb94-137cff38d00b","Type":"ContainerDied","Data":"48176717ea5f0f342b794b0f4b30e1c0ecb61e8068f766293f26a56f9fdacfbe"} Nov 03 02:20:11.648678 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.648631 5463 scope.go:122] "RemoveContainer" containerID="490121feb5482e557715f4df6bcd4c78f83135439d51a419166e1490406fd625" Nov 03 02:20:11.649008 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:20:11.648985 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"490121feb5482e557715f4df6bcd4c78f83135439d51a419166e1490406fd625\": container with ID starting with 490121feb5482e557715f4df6bcd4c78f83135439d51a419166e1490406fd625 not found: ID does not exist" containerID="490121feb5482e557715f4df6bcd4c78f83135439d51a419166e1490406fd625" Nov 03 02:20:11.649110 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.649021 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490121feb5482e557715f4df6bcd4c78f83135439d51a419166e1490406fd625"} err="failed to get container status \"490121feb5482e557715f4df6bcd4c78f83135439d51a419166e1490406fd625\": rpc error: code = NotFound desc = could not find container \"490121feb5482e557715f4df6bcd4c78f83135439d51a419166e1490406fd625\": container with ID starting with 490121feb5482e557715f4df6bcd4c78f83135439d51a419166e1490406fd625 not found: ID does not exist" Nov 03 02:20:11.649110 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.649044 5463 scope.go:122] "RemoveContainer" containerID="b9bacf70db550e4eb4dbd260269a15b28baab037ca29e0d3fb0f311a4313e9bd" Nov 03 02:20:11.663333 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.663301 5463 scope.go:122] "RemoveContainer" containerID="b9bacf70db550e4eb4dbd260269a15b28baab037ca29e0d3fb0f311a4313e9bd" Nov 03 02:20:11.663733 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:20:11.663708 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9bacf70db550e4eb4dbd260269a15b28baab037ca29e0d3fb0f311a4313e9bd\": container with ID starting with b9bacf70db550e4eb4dbd260269a15b28baab037ca29e0d3fb0f311a4313e9bd not found: ID does not exist" containerID="b9bacf70db550e4eb4dbd260269a15b28baab037ca29e0d3fb0f311a4313e9bd" Nov 03 02:20:11.663831 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.663746 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9bacf70db550e4eb4dbd260269a15b28baab037ca29e0d3fb0f311a4313e9bd"} err="failed to get container status \"b9bacf70db550e4eb4dbd260269a15b28baab037ca29e0d3fb0f311a4313e9bd\": rpc error: code = NotFound desc = could not find container \"b9bacf70db550e4eb4dbd260269a15b28baab037ca29e0d3fb0f311a4313e9bd\": container with ID starting with b9bacf70db550e4eb4dbd260269a15b28baab037ca29e0d3fb0f311a4313e9bd not found: ID does not exist" Nov 03 02:20:11.804654 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.804620 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf"] Nov 03 02:20:11.812239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.810913 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9b509-5bf69bb979-5znzf"] Nov 03 02:20:11.923414 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.923377 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x"] Nov 03 02:20:11.929277 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:11.929251 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9b509-568587dccf-8pr8x"] Nov 03 02:20:12.444579 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:12.444547 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82305fbb-af59-4971-bb94-137cff38d00b" path="/var/lib/kubelet/pods/82305fbb-af59-4971-bb94-137cff38d00b/volumes" Nov 03 02:20:12.445204 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:12.445166 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db83203f-b4b2-467f-9fb7-b105d5e89751" path="/var/lib/kubelet/pods/db83203f-b4b2-467f-9fb7-b105d5e89751/volumes" Nov 03 02:20:12.671050 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:12.671014 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:13.048711 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:20:13.048670 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:20:13.121886 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:13.121855 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:17.613266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:17.613236 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw" Nov 03 02:20:17.613668 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:17.613310 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn" Nov 03 02:20:18.219689 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:18.219621 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:23.363641 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:23.363534 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:23.748059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:23.748030 5463 generic.go:356] "Generic (PLEG): container finished" podID="d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" containerID="7b9e02876cb8942f0f9fc54b2eccdbf928cae3fd83221b37f283bb3a79523746" exitCode=0 Nov 03 02:20:23.748260 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:23.748101 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" event={"ID":"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de","Type":"ContainerDied","Data":"7b9e02876cb8942f0f9fc54b2eccdbf928cae3fd83221b37f283bb3a79523746"} Nov 03 02:20:23.748525 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:23.748512 5463 scope.go:122] "RemoveContainer" containerID="7b9e02876cb8942f0f9fc54b2eccdbf928cae3fd83221b37f283bb3a79523746" Nov 03 02:20:23.750004 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:23.749969 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:20:23.750072 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:23.750044 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:20:23.750108 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:23.750078 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:20:24.779263 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:24.779224 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" event={"ID":"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de","Type":"ContainerStarted","Data":"15724a1b56caa393d9eab37121e76db3f2d8bf36578a5dd60531f4d07982c30b"} Nov 03 02:20:25.689383 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:25.689341 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:20:25.690776 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:25.690750 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 02:20:25.690864 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:25.690801 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" podUID="d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 02:20:26.689827 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:26.689792 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 02:20:26.690253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:26.689843 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" podUID="d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 02:20:27.507688 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:27.507650 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:20:27.689647 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:27.689604 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 02:20:27.689871 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:27.689665 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" podUID="d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 02:20:28.434323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:28.434284 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:28.690446 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:28.690350 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 02:20:28.690885 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:28.690435 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" podUID="d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 02:20:29.689591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:29.689561 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 02:20:29.689777 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:29.689624 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" podUID="d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 02:20:30.694349 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:30.694316 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:20:30.695419 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:30.695394 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:20:32.196568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:32.196537 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:32.350615 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:32.350534 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:32.719945 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:32.719910 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:33.131021 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.130907 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:33.619860 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.619828 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv"] Nov 03 02:20:33.620876 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.620853 5463 cpu_manager.go:414] "containerMap: removing container" podUID="82305fbb-af59-4971-bb94-137cff38d00b" containerName="payload-pre-processing" Nov 03 02:20:33.620924 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.620880 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82305fbb-af59-4971-bb94-137cff38d00b" containerName="payload-pre-processing" Nov 03 02:20:33.620981 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.620967 5463 cpu_manager.go:414] "containerMap: removing container" podUID="db83203f-b4b2-467f-9fb7-b105d5e89751" containerName="payload-processing" Nov 03 02:20:33.621016 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.620983 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db83203f-b4b2-467f-9fb7-b105d5e89751" containerName="payload-processing" Nov 03 02:20:33.621170 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.621155 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="db83203f-b4b2-467f-9fb7-b105d5e89751" containerName="payload-processing" Nov 03 02:20:33.621222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.621181 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="82305fbb-af59-4971-bb94-137cff38d00b" containerName="payload-pre-processing" Nov 03 02:20:33.624642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.624623 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.677503 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.677468 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9bd3ab-openshift-default-dockercfg-sf9v5\"" Nov 03 02:20:33.731037 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.731005 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv"] Nov 03 02:20:33.732392 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.732364 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:33.736642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.736614 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/943f92d8-5f35-4532-a30e-41961407b68b-istiod-ca-cert\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.736725 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.736681 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-workload-certs\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.736777 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.736756 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-workload-socket\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.736834 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.736826 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/943f92d8-5f35-4532-a30e-41961407b68b-istio-podinfo\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.736941 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.736923 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/943f92d8-5f35-4532-a30e-41961407b68b-istio-token\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.736998 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.736962 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6rc6\" (UniqueName: \"kubernetes.io/projected/943f92d8-5f35-4532-a30e-41961407b68b-kube-api-access-h6rc6\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.737061 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.737038 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-istio-envoy\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.737186 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.737171 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-istio-data\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.737289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.737276 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-credential-socket\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.840992 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.840947 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-workload-certs\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.841207 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.841008 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-workload-socket\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.841207 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.841081 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/943f92d8-5f35-4532-a30e-41961407b68b-istio-podinfo\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.841207 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.841109 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/943f92d8-5f35-4532-a30e-41961407b68b-istio-token\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.841207 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.841143 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6rc6\" (UniqueName: \"kubernetes.io/projected/943f92d8-5f35-4532-a30e-41961407b68b-kube-api-access-h6rc6\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.841207 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.841172 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-istio-envoy\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.841546 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.841505 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-workload-certs\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.843672 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.843631 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-istio-data\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.843672 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.843667 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f"] Nov 03 02:20:33.843878 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.843735 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-credential-socket\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.843878 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.843794 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/943f92d8-5f35-4532-a30e-41961407b68b-istiod-ca-cert\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.844570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.844542 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/943f92d8-5f35-4532-a30e-41961407b68b-istiod-ca-cert\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.845694 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.845667 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-workload-socket\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.846481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.846441 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-istio-data\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.859480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.847316 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-credential-socket\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.859480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.848332 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f" Nov 03 02:20:33.859480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.849289 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-istio-envoy\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.860884 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.860862 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/943f92d8-5f35-4532-a30e-41961407b68b-istio-podinfo\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.898238 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.896211 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f"] Nov 03 02:20:33.910008 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.908859 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ff26f5\"" Nov 03 02:20:33.912101 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.912078 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6rc6\" (UniqueName: \"kubernetes.io/projected/943f92d8-5f35-4532-a30e-41961407b68b-kube-api-access-h6rc6\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.912687 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.912660 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/943f92d8-5f35-4532-a30e-41961407b68b-istio-token\") pod \"e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.938672 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.938650 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:33.945439 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.945417 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f\" (UID: \"e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f" Nov 03 02:20:33.945584 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.945564 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj4lw\" (UniqueName: \"kubernetes.io/projected/e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98-kube-api-access-pj4lw\") pod \"payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f\" (UID: \"e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f" Nov 03 02:20:33.978592 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:33.978563 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ff26f5-dockercfg-4d9rf\"" Nov 03 02:20:34.047501 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.047462 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj4lw\" (UniqueName: \"kubernetes.io/projected/e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98-kube-api-access-pj4lw\") pod \"payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f\" (UID: \"e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f" Nov 03 02:20:34.047672 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.047647 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f\" (UID: \"e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f" Nov 03 02:20:34.049660 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.048626 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f\" (UID: \"e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f" Nov 03 02:20:34.104860 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.104830 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj4lw\" (UniqueName: \"kubernetes.io/projected/e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98-kube-api-access-pj4lw\") pod \"payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f\" (UID: \"e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f" Nov 03 02:20:34.216605 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.216519 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f" Nov 03 02:20:34.225469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.225422 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:34.377096 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:20:34.377064 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943f92d8_5f35_4532_a30e_41961407b68b.slice/crio-6b34aafbecb6a8f183c23f6f867687e883c83fbba85517327808beb1a72af596 WatchSource:0}: Error finding container 6b34aafbecb6a8f183c23f6f867687e883c83fbba85517327808beb1a72af596: Status 404 returned error can't find the container with id 6b34aafbecb6a8f183c23f6f867687e883c83fbba85517327808beb1a72af596 Nov 03 02:20:34.379675 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.379641 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:20:34.379787 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.379728 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:20:34.379787 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.379758 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:20:34.380635 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.380612 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv"] Nov 03 02:20:34.592017 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:20:34.591984 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93d28f4_8a13_4d8a_a6a5_8d5bdb516f98.slice/crio-a79fbde7716b5054495a3bb46cb1061eefbbafceee740c9e62c9e3b081f94f32 WatchSource:0}: Error finding container a79fbde7716b5054495a3bb46cb1061eefbbafceee740c9e62c9e3b081f94f32: Status 404 returned error can't find the container with id a79fbde7716b5054495a3bb46cb1061eefbbafceee740c9e62c9e3b081f94f32 Nov 03 02:20:34.596180 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.596156 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f"] Nov 03 02:20:34.613523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.613499 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc"] Nov 03 02:20:34.617718 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.617689 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc" Nov 03 02:20:34.657857 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.657829 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq644\" (UniqueName: \"kubernetes.io/projected/39a6a726-254a-4e88-9f5e-de47741cc446-kube-api-access-hq644\") pod \"payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc\" (UID: \"39a6a726-254a-4e88-9f5e-de47741cc446\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc" Nov 03 02:20:34.658275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.657900 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a6a726-254a-4e88-9f5e-de47741cc446-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc\" (UID: \"39a6a726-254a-4e88-9f5e-de47741cc446\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc" Nov 03 02:20:34.672045 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.672017 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc"] Nov 03 02:20:34.694708 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.694670 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q"] Nov 03 02:20:34.759903 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.759571 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq644\" (UniqueName: \"kubernetes.io/projected/39a6a726-254a-4e88-9f5e-de47741cc446-kube-api-access-hq644\") pod \"payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc\" (UID: \"39a6a726-254a-4e88-9f5e-de47741cc446\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc" Nov 03 02:20:34.759903 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.759661 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a6a726-254a-4e88-9f5e-de47741cc446-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc\" (UID: \"39a6a726-254a-4e88-9f5e-de47741cc446\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc" Nov 03 02:20:34.760597 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.760568 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a6a726-254a-4e88-9f5e-de47741cc446-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc\" (UID: \"39a6a726-254a-4e88-9f5e-de47741cc446\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc" Nov 03 02:20:34.788492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.788461 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg"] Nov 03 02:20:34.816529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.816505 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq644\" (UniqueName: \"kubernetes.io/projected/39a6a726-254a-4e88-9f5e-de47741cc446-kube-api-access-hq644\") pod \"payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc\" (UID: \"39a6a726-254a-4e88-9f5e-de47741cc446\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc" Nov 03 02:20:34.903723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.903686 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" event={"ID":"943f92d8-5f35-4532-a30e-41961407b68b","Type":"ContainerStarted","Data":"7b628e4470c5c1a7882801a7696f5384c05a11ac808d4fe7a016b03b729a03a7"} Nov 03 02:20:34.903930 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.903752 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" event={"ID":"943f92d8-5f35-4532-a30e-41961407b68b","Type":"ContainerStarted","Data":"6b34aafbecb6a8f183c23f6f867687e883c83fbba85517327808beb1a72af596"} Nov 03 02:20:34.905054 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.905026 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f" event={"ID":"e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98","Type":"ContainerStarted","Data":"a79fbde7716b5054495a3bb46cb1061eefbbafceee740c9e62c9e3b081f94f32"} Nov 03 02:20:34.930631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.930605 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc" Nov 03 02:20:34.939515 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:34.939493 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:35.084604 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:35.084540 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" podStartSLOduration=2.084518655 podStartE2EDuration="2.084518655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:20:35.079918204 +0000 UTC m=+1716.764027359" watchObservedRunningTime="2026-11-03 02:20:35.084518655 +0000 UTC m=+1716.768627811" Nov 03 02:20:35.087043 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:35.086098 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:35.233759 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:35.233660 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:35.322081 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:20:35.322049 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a6a726_254a_4e88_9f5e_de47741cc446.slice/crio-bf8866d2b8904cd2931439ec1c36869bebb4f4142f053923bfccb435d19463d2 WatchSource:0}: Error finding container bf8866d2b8904cd2931439ec1c36869bebb4f4142f053923bfccb435d19463d2: Status 404 returned error can't find the container with id bf8866d2b8904cd2931439ec1c36869bebb4f4142f053923bfccb435d19463d2 Nov 03 02:20:35.325720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:35.325632 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc"] Nov 03 02:20:35.384545 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:35.384513 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:35.913323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:35.913286 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc" event={"ID":"39a6a726-254a-4e88-9f5e-de47741cc446","Type":"ContainerStarted","Data":"bf8866d2b8904cd2931439ec1c36869bebb4f4142f053923bfccb435d19463d2"} Nov 03 02:20:35.944655 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:35.944626 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:36.921422 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:36.921391 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:36.922250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:36.922226 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:20:38.883942 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:38.883909 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:39.049156 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:39.049090 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:39.249441 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:39.249403 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:39.695289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:39.695228 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q" podUID="b9f81dca-b2c4-4d87-afc1-08db93b24e6b" containerName="payload-processing" containerID="cri-o://0a80d49f8de0ea770c2e3aafdb89914e102f1ab4cdb9e41923d6f051474a70f0" gracePeriod=25 Nov 03 02:20:39.789908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:39.789860 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg" podUID="c5089d5c-3c50-40d7-a89e-90946d0404e3" containerName="payload-pre-processing" containerID="cri-o://7a0090a7e3b5f5108a33324e9dd4a1ad6533800dd51796edb73ce9d175adc142" gracePeriod=25 Nov 03 02:20:39.952796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:39.951769 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc" event={"ID":"39a6a726-254a-4e88-9f5e-de47741cc446","Type":"ContainerStarted","Data":"2cf604e9081f9c20b8d5dc2cc7b4593f95569dea64de99f64ef067209a991712"} Nov 03 02:20:39.952796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:39.951858 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc" Nov 03 02:20:39.956025 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:39.955122 5463 generic.go:356] "Generic (PLEG): container finished" podID="c5089d5c-3c50-40d7-a89e-90946d0404e3" containerID="7a0090a7e3b5f5108a33324e9dd4a1ad6533800dd51796edb73ce9d175adc142" exitCode=0 Nov 03 02:20:39.956025 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:39.955225 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg" event={"ID":"c5089d5c-3c50-40d7-a89e-90946d0404e3","Type":"ContainerDied","Data":"7a0090a7e3b5f5108a33324e9dd4a1ad6533800dd51796edb73ce9d175adc142"} Nov 03 02:20:39.957073 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:39.957050 5463 generic.go:356] "Generic (PLEG): container finished" podID="b9f81dca-b2c4-4d87-afc1-08db93b24e6b" containerID="0a80d49f8de0ea770c2e3aafdb89914e102f1ab4cdb9e41923d6f051474a70f0" exitCode=0 Nov 03 02:20:39.957223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:39.957177 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q" event={"ID":"b9f81dca-b2c4-4d87-afc1-08db93b24e6b","Type":"ContainerDied","Data":"0a80d49f8de0ea770c2e3aafdb89914e102f1ab4cdb9e41923d6f051474a70f0"} Nov 03 02:20:39.959007 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:39.958983 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f" event={"ID":"e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98","Type":"ContainerStarted","Data":"3bae839164142af27a779d0c6c56f5c2e016160916b7bec8bd954609afdd9aec"} Nov 03 02:20:39.959146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:39.959128 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f" Nov 03 02:20:40.068952 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.068170 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc" podStartSLOduration=1.7444054740000001 podStartE2EDuration="6.068150739s" totalImagesPullingTime="4.323745265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:20:40.066553748 +0000 UTC m=+1721.750662904" watchObservedRunningTime="2026-11-03 02:20:40.068150739 +0000 UTC m=+1721.752259895" Nov 03 02:20:40.077053 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.077029 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q" Nov 03 02:20:40.115490 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.115471 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg" Nov 03 02:20:40.144294 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.144269 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9f81dca-b2c4-4d87-afc1-08db93b24e6b-kube-api-access-8v6nc\" (UniqueName: \"kubernetes.io/projected/b9f81dca-b2c4-4d87-afc1-08db93b24e6b-kube-api-access-8v6nc\") pod \"b9f81dca-b2c4-4d87-afc1-08db93b24e6b\" (UID: \"b9f81dca-b2c4-4d87-afc1-08db93b24e6b\") " Nov 03 02:20:40.144433 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.144354 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9f81dca-b2c4-4d87-afc1-08db93b24e6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f81dca-b2c4-4d87-afc1-08db93b24e6b-plugins-config-volume\") pod \"b9f81dca-b2c4-4d87-afc1-08db93b24e6b\" (UID: \"b9f81dca-b2c4-4d87-afc1-08db93b24e6b\") " Nov 03 02:20:40.145079 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.145057 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f81dca-b2c4-4d87-afc1-08db93b24e6b-plugins-config-volume" pod "b9f81dca-b2c4-4d87-afc1-08db93b24e6b" (UID: "b9f81dca-b2c4-4d87-afc1-08db93b24e6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:20:40.147484 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.147458 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f81dca-b2c4-4d87-afc1-08db93b24e6b-kube-api-access-8v6nc" pod "b9f81dca-b2c4-4d87-afc1-08db93b24e6b" (UID: "b9f81dca-b2c4-4d87-afc1-08db93b24e6b"). InnerVolumeSpecName "kube-api-access-8v6nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:20:40.245816 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.245739 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5089d5c-3c50-40d7-a89e-90946d0404e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5089d5c-3c50-40d7-a89e-90946d0404e3-plugins-config-volume\") pod \"c5089d5c-3c50-40d7-a89e-90946d0404e3\" (UID: \"c5089d5c-3c50-40d7-a89e-90946d0404e3\") " Nov 03 02:20:40.245816 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.245798 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5089d5c-3c50-40d7-a89e-90946d0404e3-kube-api-access-ltxjs\" (UniqueName: \"kubernetes.io/projected/c5089d5c-3c50-40d7-a89e-90946d0404e3-kube-api-access-ltxjs\") pod \"c5089d5c-3c50-40d7-a89e-90946d0404e3\" (UID: \"c5089d5c-3c50-40d7-a89e-90946d0404e3\") " Nov 03 02:20:40.246128 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.246105 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5089d5c-3c50-40d7-a89e-90946d0404e3-plugins-config-volume" pod "c5089d5c-3c50-40d7-a89e-90946d0404e3" (UID: "c5089d5c-3c50-40d7-a89e-90946d0404e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:20:40.246328 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.246309 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v6nc\" (UniqueName: \"kubernetes.io/projected/b9f81dca-b2c4-4d87-afc1-08db93b24e6b-kube-api-access-8v6nc\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:40.246381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.246332 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f81dca-b2c4-4d87-afc1-08db93b24e6b-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:40.246381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.246343 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5089d5c-3c50-40d7-a89e-90946d0404e3-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:40.248332 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.248307 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5089d5c-3c50-40d7-a89e-90946d0404e3-kube-api-access-ltxjs" pod "c5089d5c-3c50-40d7-a89e-90946d0404e3" (UID: "c5089d5c-3c50-40d7-a89e-90946d0404e3"). InnerVolumeSpecName "kube-api-access-ltxjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:20:40.301123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.300862 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f" podStartSLOduration=2.211654752 podStartE2EDuration="7.30084655s" totalImagesPullingTime="5.089191798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:20:40.184332418 +0000 UTC m=+1721.868441577" watchObservedRunningTime="2026-11-03 02:20:40.30084655 +0000 UTC m=+1721.984955706" Nov 03 02:20:40.315714 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.315682 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:40.347873 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.347835 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltxjs\" (UniqueName: \"kubernetes.io/projected/c5089d5c-3c50-40d7-a89e-90946d0404e3-kube-api-access-ltxjs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:40.968983 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.968894 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg" Nov 03 02:20:40.968983 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.968901 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg" event={"ID":"c5089d5c-3c50-40d7-a89e-90946d0404e3","Type":"ContainerDied","Data":"2e65de484ec09a6896a540ab602c2cc695d53dce1b04332b19aa754390a92582"} Nov 03 02:20:40.969527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.969031 5463 scope.go:122] "RemoveContainer" containerID="7a0090a7e3b5f5108a33324e9dd4a1ad6533800dd51796edb73ce9d175adc142" Nov 03 02:20:40.970570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.970546 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q" Nov 03 02:20:40.970709 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.970623 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q" event={"ID":"b9f81dca-b2c4-4d87-afc1-08db93b24e6b","Type":"ContainerDied","Data":"4481dcdbe2d65f9e15b542604b9cf28191968ec8bf7d962de22153ccdb49d0dc"} Nov 03 02:20:40.981877 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:40.981855 5463 scope.go:122] "RemoveContainer" containerID="0a80d49f8de0ea770c2e3aafdb89914e102f1ab4cdb9e41923d6f051474a70f0" Nov 03 02:20:41.134128 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:41.134098 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q"] Nov 03 02:20:41.184913 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:41.184878 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f66575-67577ffbff-ghb4q"] Nov 03 02:20:41.297224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:41.297128 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg"] Nov 03 02:20:41.300108 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:41.300086 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-7b495bfbf9-k68fg"] Nov 03 02:20:41.516709 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:41.516656 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:41.922882 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:41.922850 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g"] Nov 03 02:20:42.030434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:42.030405 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt"] Nov 03 02:20:42.297775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:42.297683 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:42.447610 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:42.447574 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9f81dca-b2c4-4d87-afc1-08db93b24e6b" path="/var/lib/kubelet/pods/b9f81dca-b2c4-4d87-afc1-08db93b24e6b/volumes" Nov 03 02:20:42.448253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:42.448225 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5089d5c-3c50-40d7-a89e-90946d0404e3" path="/var/lib/kubelet/pods/c5089d5c-3c50-40d7-a89e-90946d0404e3/volumes" Nov 03 02:20:42.449825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:42.449799 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:43.034886 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:20:43.032500 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:20:43.395835 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.395129 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd"] Nov 03 02:20:43.396111 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.396085 5463 cpu_manager.go:414] "containerMap: removing container" podUID="b9f81dca-b2c4-4d87-afc1-08db93b24e6b" containerName="payload-processing" Nov 03 02:20:43.396258 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.396115 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f81dca-b2c4-4d87-afc1-08db93b24e6b" containerName="payload-processing" Nov 03 02:20:43.396258 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.396172 5463 cpu_manager.go:414] "containerMap: removing container" podUID="c5089d5c-3c50-40d7-a89e-90946d0404e3" containerName="payload-pre-processing" Nov 03 02:20:43.396258 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.396181 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5089d5c-3c50-40d7-a89e-90946d0404e3" containerName="payload-pre-processing" Nov 03 02:20:43.396427 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.396384 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f81dca-b2c4-4d87-afc1-08db93b24e6b" containerName="payload-processing" Nov 03 02:20:43.396427 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.396403 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5089d5c-3c50-40d7-a89e-90946d0404e3" containerName="payload-pre-processing" Nov 03 02:20:43.400262 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.400243 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.494402 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.494358 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-workload-socket\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.494650 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.494631 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-data\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.494800 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.494787 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-envoy\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.494960 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.494948 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-workload-certs\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.495070 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.495058 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-credential-socket\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.495171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.495159 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-podinfo\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.495316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.495302 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istiod-ca-cert\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.495417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.495404 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-token\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.495609 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.495597 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gh6b\" (UniqueName: \"kubernetes.io/projected/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-kube-api-access-7gh6b\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.495847 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.495834 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd"] Nov 03 02:20:43.512238 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.504938 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ad4591-openshift-default-dockercfg-gczw6\"" Nov 03 02:20:43.512238 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.506275 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:43.597009 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.596977 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gh6b\" (UniqueName: \"kubernetes.io/projected/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-kube-api-access-7gh6b\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.597474 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.597444 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-workload-socket\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.597630 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.597483 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-data\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.597630 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.597530 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-envoy\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.597630 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.597581 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-workload-certs\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.597630 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.597606 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-credential-socket\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.597630 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.597626 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-podinfo\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.597979 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.597660 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istiod-ca-cert\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.597979 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.597683 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-token\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.598639 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.598410 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-workload-socket\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.598639 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.598566 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-workload-certs\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.598862 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.598833 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-data\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.599313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.599284 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-credential-socket\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.599715 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.599690 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istiod-ca-cert\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.602387 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.602363 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-envoy\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.604804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.604782 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-podinfo\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.654697 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.654619 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-token\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.654838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.654795 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gh6b\" (UniqueName: \"kubernetes.io/projected/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-kube-api-access-7gh6b\") pod \"e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.715216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.713520 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:43.756093 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:43.756061 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:44.167517 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:44.167481 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd"] Nov 03 02:20:44.198234 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:20:44.198207 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod255cfc8e_f969_4678_b7b0_b37a4bc8dd4e.slice/crio-e49dabc85c928a0e7c58803995609a2e2834459b2a645a89d3261249b7e0bd2a WatchSource:0}: Error finding container e49dabc85c928a0e7c58803995609a2e2834459b2a645a89d3261249b7e0bd2a: Status 404 returned error can't find the container with id e49dabc85c928a0e7c58803995609a2e2834459b2a645a89d3261249b7e0bd2a Nov 03 02:20:44.201059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:44.201024 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:20:44.201177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:44.201093 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:20:44.201177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:44.201123 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:20:45.028440 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.028404 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" event={"ID":"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e","Type":"ContainerStarted","Data":"fe60cace8e3b1fbf70992ffacac504bcb16db23fdbe1286a97c2c387bb6b77ae"} Nov 03 02:20:45.028599 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.028467 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" event={"ID":"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e","Type":"ContainerStarted","Data":"e49dabc85c928a0e7c58803995609a2e2834459b2a645a89d3261249b7e0bd2a"} Nov 03 02:20:45.198450 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.198354 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" podStartSLOduration=2.198338726 podStartE2EDuration="2.198338726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:20:45.145174052 +0000 UTC m=+1726.829283207" watchObservedRunningTime="2026-11-03 02:20:45.198338726 +0000 UTC m=+1726.882447878" Nov 03 02:20:45.201738 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.201711 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw"] Nov 03 02:20:45.206079 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.206056 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" Nov 03 02:20:45.212273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.212249 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw"] Nov 03 02:20:45.258970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.258923 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5ef2c8-serving-cert\"" Nov 03 02:20:45.319498 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.319459 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw\" (UID: \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" Nov 03 02:20:45.319694 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.319603 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-metrics-certs\") pod \"maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw\" (UID: \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" Nov 03 02:20:45.319694 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.319637 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw8jc\" (UniqueName: \"kubernetes.io/projected/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-kube-api-access-nw8jc\") pod \"maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw\" (UID: \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" Nov 03 02:20:45.319791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.319739 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw\" (UID: \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" Nov 03 02:20:45.366139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.366106 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:45.421435 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.421400 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-metrics-certs\") pod \"maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw\" (UID: \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" Nov 03 02:20:45.421641 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.421441 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw8jc\" (UniqueName: \"kubernetes.io/projected/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-kube-api-access-nw8jc\") pod \"maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw\" (UID: \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" Nov 03 02:20:45.421641 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.421515 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw\" (UID: \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" Nov 03 02:20:45.421641 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.421617 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw\" (UID: \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" Nov 03 02:20:45.421808 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:20:45.421774 5463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-serving-cert: secret "maas-api-e2e-ipp-b-5ef2c8-serving-cert" not found Nov 03 02:20:45.421878 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:20:45.421865 5463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-maas-api-tls podName:feb8fd27-a4ea-41d8-ba16-69279ff9e85b nodeName:}" failed. No retries permitted until 2026-11-03 02:20:45.9218431 +0000 UTC m=+1727.605952234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-maas-api-tls") pod "maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" (UID: "feb8fd27-a4ea-41d8-ba16-69279ff9e85b") : secret "maas-api-e2e-ipp-b-5ef2c8-serving-cert" not found Nov 03 02:20:45.422173 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.422155 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw\" (UID: \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" Nov 03 02:20:45.424043 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.424025 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-metrics-certs\") pod \"maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw\" (UID: \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" Nov 03 02:20:45.479360 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.479286 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw8jc\" (UniqueName: \"kubernetes.io/projected/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-kube-api-access-nw8jc\") pod \"maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw\" (UID: \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" Nov 03 02:20:45.522357 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.522330 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:45.670452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.670411 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:45.714809 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.714743 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:45.720396 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.720373 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:45.836721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.836649 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:45.927757 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.927718 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw\" (UID: \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" Nov 03 02:20:45.931753 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:45.931720 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw\" (UID: \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" Nov 03 02:20:46.012175 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.012141 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:46.035078 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.035047 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:46.036023 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.036001 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:20:46.129809 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.129723 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" Nov 03 02:20:46.241560 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.240508 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf"] Nov 03 02:20:46.248259 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.247006 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf" Nov 03 02:20:46.296592 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.296403 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf"] Nov 03 02:20:46.305373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.305343 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9bd3ab\"" Nov 03 02:20:46.305544 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.305499 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9bd3ab-dockercfg-hhpmd\"" Nov 03 02:20:46.372161 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.372118 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwnzd\" (UniqueName: \"kubernetes.io/projected/aaff3f9e-3094-4e97-863e-c832964b6a61-kube-api-access-pwnzd\") pod \"payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf\" (UID: \"aaff3f9e-3094-4e97-863e-c832964b6a61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf" Nov 03 02:20:46.372563 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.372535 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaff3f9e-3094-4e97-863e-c832964b6a61-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf\" (UID: \"aaff3f9e-3094-4e97-863e-c832964b6a61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf" Nov 03 02:20:46.386019 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.385933 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk"] Nov 03 02:20:46.404972 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.403397 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk" Nov 03 02:20:46.446848 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.446823 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk"] Nov 03 02:20:46.474662 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.474632 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwnzd\" (UniqueName: \"kubernetes.io/projected/aaff3f9e-3094-4e97-863e-c832964b6a61-kube-api-access-pwnzd\") pod \"payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf\" (UID: \"aaff3f9e-3094-4e97-863e-c832964b6a61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf" Nov 03 02:20:46.474819 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.474806 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaff3f9e-3094-4e97-863e-c832964b6a61-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf\" (UID: \"aaff3f9e-3094-4e97-863e-c832964b6a61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf" Nov 03 02:20:46.475726 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.475700 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaff3f9e-3094-4e97-863e-c832964b6a61-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf\" (UID: \"aaff3f9e-3094-4e97-863e-c832964b6a61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf" Nov 03 02:20:46.531813 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.531785 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwnzd\" (UniqueName: \"kubernetes.io/projected/aaff3f9e-3094-4e97-863e-c832964b6a61-kube-api-access-pwnzd\") pod \"payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf\" (UID: \"aaff3f9e-3094-4e97-863e-c832964b6a61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf" Nov 03 02:20:46.546656 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:20:46.546623 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb8fd27_a4ea_41d8_ba16_69279ff9e85b.slice/crio-822529bc576f96043af79ca09c87850947748f19badae127dba91f5f19eec707 WatchSource:0}: Error finding container 822529bc576f96043af79ca09c87850947748f19badae127dba91f5f19eec707: Status 404 returned error can't find the container with id 822529bc576f96043af79ca09c87850947748f19badae127dba91f5f19eec707 Nov 03 02:20:46.551718 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.551467 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw"] Nov 03 02:20:46.568052 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.568032 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf" Nov 03 02:20:46.576596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.576564 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2152640c-b513-4397-b448-dbd5199d6aaa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk\" (UID: \"2152640c-b513-4397-b448-dbd5199d6aaa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk" Nov 03 02:20:46.576699 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.576635 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dvlq\" (UniqueName: \"kubernetes.io/projected/2152640c-b513-4397-b448-dbd5199d6aaa-kube-api-access-5dvlq\") pod \"payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk\" (UID: \"2152640c-b513-4397-b448-dbd5199d6aaa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk" Nov 03 02:20:46.678202 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.678152 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2152640c-b513-4397-b448-dbd5199d6aaa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk\" (UID: \"2152640c-b513-4397-b448-dbd5199d6aaa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk" Nov 03 02:20:46.678353 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.678251 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dvlq\" (UniqueName: \"kubernetes.io/projected/2152640c-b513-4397-b448-dbd5199d6aaa-kube-api-access-5dvlq\") pod \"payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk\" (UID: \"2152640c-b513-4397-b448-dbd5199d6aaa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk" Nov 03 02:20:46.679982 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.679845 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2152640c-b513-4397-b448-dbd5199d6aaa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk\" (UID: \"2152640c-b513-4397-b448-dbd5199d6aaa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk" Nov 03 02:20:46.735756 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.735723 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dvlq\" (UniqueName: \"kubernetes.io/projected/2152640c-b513-4397-b448-dbd5199d6aaa-kube-api-access-5dvlq\") pod \"payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk\" (UID: \"2152640c-b513-4397-b448-dbd5199d6aaa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk" Nov 03 02:20:46.923771 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.923693 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g" podUID="fa734878-76ae-43cd-b47b-d5752ded7a8b" containerName="payload-processing" containerID="cri-o://f7b49f94752a2b0a31f2761f4d519ec8632d5b33ec97326636586abd197b5093" gracePeriod=25 Nov 03 02:20:46.937596 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:20:46.937571 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaff3f9e_3094_4e97_863e_c832964b6a61.slice/crio-80c2f89cc3283f433582184ff68c761029c7d37358fced16966388a07272473b WatchSource:0}: Error finding container 80c2f89cc3283f433582184ff68c761029c7d37358fced16966388a07272473b: Status 404 returned error can't find the container with id 80c2f89cc3283f433582184ff68c761029c7d37358fced16966388a07272473b Nov 03 02:20:46.942232 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:46.942203 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf"] Nov 03 02:20:47.024528 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.024503 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk" Nov 03 02:20:47.031689 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.031664 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt" podUID="cd50e030-93e4-41ab-afc4-21f098a66911" containerName="payload-pre-processing" containerID="cri-o://725601e93cabfdb737864ad0920c1feee88553145440ede7e4b8168a6c45c2ac" gracePeriod=25 Nov 03 02:20:47.047708 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.047670 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf" event={"ID":"aaff3f9e-3094-4e97-863e-c832964b6a61","Type":"ContainerStarted","Data":"80c2f89cc3283f433582184ff68c761029c7d37358fced16966388a07272473b"} Nov 03 02:20:47.049679 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.049651 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" event={"ID":"feb8fd27-a4ea-41d8-ba16-69279ff9e85b","Type":"ContainerStarted","Data":"822529bc576f96043af79ca09c87850947748f19badae127dba91f5f19eec707"} Nov 03 02:20:47.053535 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.053498 5463 generic.go:356] "Generic (PLEG): container finished" podID="fa734878-76ae-43cd-b47b-d5752ded7a8b" containerID="f7b49f94752a2b0a31f2761f4d519ec8632d5b33ec97326636586abd197b5093" exitCode=0 Nov 03 02:20:47.053654 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.053558 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g" event={"ID":"fa734878-76ae-43cd-b47b-d5752ded7a8b","Type":"ContainerDied","Data":"f7b49f94752a2b0a31f2761f4d519ec8632d5b33ec97326636586abd197b5093"} Nov 03 02:20:47.236041 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.235669 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g" Nov 03 02:20:47.348035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.347945 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt" Nov 03 02:20:47.388830 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.388790 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa734878-76ae-43cd-b47b-d5752ded7a8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa734878-76ae-43cd-b47b-d5752ded7a8b-plugins-config-volume\") pod \"fa734878-76ae-43cd-b47b-d5752ded7a8b\" (UID: \"fa734878-76ae-43cd-b47b-d5752ded7a8b\") " Nov 03 02:20:47.389025 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.388856 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa734878-76ae-43cd-b47b-d5752ded7a8b-kube-api-access-mzlmm\" (UniqueName: \"kubernetes.io/projected/fa734878-76ae-43cd-b47b-d5752ded7a8b-kube-api-access-mzlmm\") pod \"fa734878-76ae-43cd-b47b-d5752ded7a8b\" (UID: \"fa734878-76ae-43cd-b47b-d5752ded7a8b\") " Nov 03 02:20:47.390222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.390173 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa734878-76ae-43cd-b47b-d5752ded7a8b-plugins-config-volume" pod "fa734878-76ae-43cd-b47b-d5752ded7a8b" (UID: "fa734878-76ae-43cd-b47b-d5752ded7a8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:20:47.392370 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.392335 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa734878-76ae-43cd-b47b-d5752ded7a8b-kube-api-access-mzlmm" pod "fa734878-76ae-43cd-b47b-d5752ded7a8b" (UID: "fa734878-76ae-43cd-b47b-d5752ded7a8b"). InnerVolumeSpecName "kube-api-access-mzlmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:20:47.413302 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:20:47.413276 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2152640c_b513_4397_b448_dbd5199d6aaa.slice/crio-8d83353e3a60c001ee2d3f087e471c9e82be8ad64c3c7fb3c3b47ec64bf631b7 WatchSource:0}: Error finding container 8d83353e3a60c001ee2d3f087e471c9e82be8ad64c3c7fb3c3b47ec64bf631b7: Status 404 returned error can't find the container with id 8d83353e3a60c001ee2d3f087e471c9e82be8ad64c3c7fb3c3b47ec64bf631b7 Nov 03 02:20:47.414257 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.414111 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk"] Nov 03 02:20:47.490847 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.490819 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd50e030-93e4-41ab-afc4-21f098a66911-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd50e030-93e4-41ab-afc4-21f098a66911-plugins-config-volume\") pod \"cd50e030-93e4-41ab-afc4-21f098a66911\" (UID: \"cd50e030-93e4-41ab-afc4-21f098a66911\") " Nov 03 02:20:47.491006 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.490866 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd50e030-93e4-41ab-afc4-21f098a66911-kube-api-access-td9l8\" (UniqueName: \"kubernetes.io/projected/cd50e030-93e4-41ab-afc4-21f098a66911-kube-api-access-td9l8\") pod \"cd50e030-93e4-41ab-afc4-21f098a66911\" (UID: \"cd50e030-93e4-41ab-afc4-21f098a66911\") " Nov 03 02:20:47.491283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.491254 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd50e030-93e4-41ab-afc4-21f098a66911-plugins-config-volume" pod "cd50e030-93e4-41ab-afc4-21f098a66911" (UID: "cd50e030-93e4-41ab-afc4-21f098a66911"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:20:47.491481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.491461 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd50e030-93e4-41ab-afc4-21f098a66911-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:47.491588 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.491489 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa734878-76ae-43cd-b47b-d5752ded7a8b-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:47.491588 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.491506 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzlmm\" (UniqueName: \"kubernetes.io/projected/fa734878-76ae-43cd-b47b-d5752ded7a8b-kube-api-access-mzlmm\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:47.493206 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.493169 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd50e030-93e4-41ab-afc4-21f098a66911-kube-api-access-td9l8" pod "cd50e030-93e4-41ab-afc4-21f098a66911" (UID: "cd50e030-93e4-41ab-afc4-21f098a66911"). InnerVolumeSpecName "kube-api-access-td9l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:20:47.593313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:47.593276 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td9l8\" (UniqueName: \"kubernetes.io/projected/cd50e030-93e4-41ab-afc4-21f098a66911-kube-api-access-td9l8\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:48.063648 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.063552 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf" event={"ID":"aaff3f9e-3094-4e97-863e-c832964b6a61","Type":"ContainerStarted","Data":"130b1331525bb518e1ed30ee98f9a62bea276527537cfcd117e7030ff767628e"} Nov 03 02:20:48.063648 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.063619 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf" Nov 03 02:20:48.064845 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.064819 5463 generic.go:356] "Generic (PLEG): container finished" podID="cd50e030-93e4-41ab-afc4-21f098a66911" containerID="725601e93cabfdb737864ad0920c1feee88553145440ede7e4b8168a6c45c2ac" exitCode=0 Nov 03 02:20:48.064924 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.064850 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt" event={"ID":"cd50e030-93e4-41ab-afc4-21f098a66911","Type":"ContainerDied","Data":"725601e93cabfdb737864ad0920c1feee88553145440ede7e4b8168a6c45c2ac"} Nov 03 02:20:48.064924 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.064895 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt" event={"ID":"cd50e030-93e4-41ab-afc4-21f098a66911","Type":"ContainerDied","Data":"e374370690594d8cf21bfa85589b512df6bdd9d6d595026f2d6234f707242aaf"} Nov 03 02:20:48.064924 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.064856 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt" Nov 03 02:20:48.064924 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.064921 5463 scope.go:122] "RemoveContainer" containerID="725601e93cabfdb737864ad0920c1feee88553145440ede7e4b8168a6c45c2ac" Nov 03 02:20:48.071364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.071331 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk" event={"ID":"2152640c-b513-4397-b448-dbd5199d6aaa","Type":"ContainerStarted","Data":"caf08aa6b8709b7bb5945061f630aca74b5ec629c4ca0f664655cb6b182a8c38"} Nov 03 02:20:48.071498 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.071390 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk" event={"ID":"2152640c-b513-4397-b448-dbd5199d6aaa","Type":"ContainerStarted","Data":"8d83353e3a60c001ee2d3f087e471c9e82be8ad64c3c7fb3c3b47ec64bf631b7"} Nov 03 02:20:48.071498 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.071423 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk" Nov 03 02:20:48.073417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.073389 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" event={"ID":"feb8fd27-a4ea-41d8-ba16-69279ff9e85b","Type":"ContainerStarted","Data":"509014614ad31824df579c81e33a136482ca54b36b2c52f493b471242d292d6d"} Nov 03 02:20:48.073557 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.073505 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" Nov 03 02:20:48.074989 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.074968 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g" Nov 03 02:20:48.075086 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.075003 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g" event={"ID":"fa734878-76ae-43cd-b47b-d5752ded7a8b","Type":"ContainerDied","Data":"8d6427c396d0efe37a647935f99ce5e5ce43fee3528ac0014010fcd9173ba11c"} Nov 03 02:20:48.082502 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.082474 5463 scope.go:122] "RemoveContainer" containerID="725601e93cabfdb737864ad0920c1feee88553145440ede7e4b8168a6c45c2ac" Nov 03 02:20:48.082767 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:20:48.082746 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"725601e93cabfdb737864ad0920c1feee88553145440ede7e4b8168a6c45c2ac\": container with ID starting with 725601e93cabfdb737864ad0920c1feee88553145440ede7e4b8168a6c45c2ac not found: ID does not exist" containerID="725601e93cabfdb737864ad0920c1feee88553145440ede7e4b8168a6c45c2ac" Nov 03 02:20:48.082828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.082778 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"725601e93cabfdb737864ad0920c1feee88553145440ede7e4b8168a6c45c2ac"} err="failed to get container status \"725601e93cabfdb737864ad0920c1feee88553145440ede7e4b8168a6c45c2ac\": rpc error: code = NotFound desc = could not find container \"725601e93cabfdb737864ad0920c1feee88553145440ede7e4b8168a6c45c2ac\": container with ID starting with 725601e93cabfdb737864ad0920c1feee88553145440ede7e4b8168a6c45c2ac not found: ID does not exist" Nov 03 02:20:48.082828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.082798 5463 scope.go:122] "RemoveContainer" containerID="f7b49f94752a2b0a31f2761f4d519ec8632d5b33ec97326636586abd197b5093" Nov 03 02:20:48.179919 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.179870 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf" podStartSLOduration=2.179856282 podStartE2EDuration="2.179856282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:20:48.177333829 +0000 UTC m=+1729.861442984" watchObservedRunningTime="2026-11-03 02:20:48.179856282 +0000 UTC m=+1729.863965436" Nov 03 02:20:48.294931 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.294870 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" podStartSLOduration=2.179021901 podStartE2EDuration="3.294850901s" totalImagesPullingTime="1.115829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:20:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:20:48.294375695 +0000 UTC m=+1729.978484851" watchObservedRunningTime="2026-11-03 02:20:48.294850901 +0000 UTC m=+1729.978960056" Nov 03 02:20:48.410686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.410636 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk" podStartSLOduration=2.410620209 podStartE2EDuration="2.410620209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:20:48.407367969 +0000 UTC m=+1730.091477124" watchObservedRunningTime="2026-11-03 02:20:48.410620209 +0000 UTC m=+1730.094729365" Nov 03 02:20:48.533333 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.533280 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g"] Nov 03 02:20:48.537966 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.537935 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-58fdffd5fc-gcf6g"] Nov 03 02:20:48.635297 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.635265 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt"] Nov 03 02:20:48.637451 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.637431 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-84b6787bc9-p2rbt"] Nov 03 02:20:48.844583 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:48.844490 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:49.055364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:49.055332 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:49.208425 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:49.208389 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:50.443799 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:50.443763 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd50e030-93e4-41ab-afc4-21f098a66911" path="/var/lib/kubelet/pods/cd50e030-93e4-41ab-afc4-21f098a66911/volumes" Nov 03 02:20:50.444309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:50.444162 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa734878-76ae-43cd-b47b-d5752ded7a8b" path="/var/lib/kubelet/pods/fa734878-76ae-43cd-b47b-d5752ded7a8b/volumes" Nov 03 02:20:50.665637 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:50.665606 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:50.973289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:50.973261 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f" Nov 03 02:20:50.973500 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:50.973344 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc" Nov 03 02:20:53.668523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:53.668481 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc"] Nov 03 02:20:53.669022 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:53.668822 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" podUID="9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" containerName="istio-proxy" containerID="cri-o://4503c097403c59562ca7ae1334e8a7617af7cfe4f49c02a73c63404faedcfffc" gracePeriod=30 Nov 03 02:20:53.714833 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:53.714804 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46"] Nov 03 02:20:53.715027 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:53.715007 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" podUID="5673c3cf-3606-42b2-968b-8ccec7e0233a" containerName="istio-proxy" containerID="cri-o://34bc20f65d467ba0d8ac2f5ca61e960260362053807362858bfbeffb8f54c51f" gracePeriod=30 Nov 03 02:20:54.090702 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:54.090617 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" Nov 03 02:20:55.282837 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:55.282797 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:55.456914 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:55.456797 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:55.637822 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:55.637777 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:55.670166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:55.670125 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5"] Nov 03 02:20:55.671125 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:55.671099 5463 cpu_manager.go:414] "containerMap: removing container" podUID="cd50e030-93e4-41ab-afc4-21f098a66911" containerName="payload-pre-processing" Nov 03 02:20:55.671125 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:55.671124 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd50e030-93e4-41ab-afc4-21f098a66911" containerName="payload-pre-processing" Nov 03 02:20:55.671349 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:55.671160 5463 cpu_manager.go:414] "containerMap: removing container" podUID="fa734878-76ae-43cd-b47b-d5752ded7a8b" containerName="payload-processing" Nov 03 02:20:55.671349 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:55.671168 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa734878-76ae-43cd-b47b-d5752ded7a8b" containerName="payload-processing" Nov 03 02:20:55.671459 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:55.671386 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa734878-76ae-43cd-b47b-d5752ded7a8b" containerName="payload-processing" Nov 03 02:20:55.671459 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:55.671412 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd50e030-93e4-41ab-afc4-21f098a66911" containerName="payload-pre-processing" Nov 03 02:20:55.676873 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:55.676852 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5" Nov 03 02:20:55.724349 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:55.724291 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5"] Nov 03 02:20:55.787211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:55.779535 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ad4591-dockercfg-sthz8\"" Nov 03 02:20:55.787211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:55.779637 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ad4591\"" Nov 03 02:20:55.902538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:55.902505 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686"] Nov 03 02:20:55.912283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:55.908760 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686" Nov 03 02:20:55.918555 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:55.918464 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9qh5\" (UniqueName: \"kubernetes.io/projected/86e79ca8-6f1f-421d-b8f8-343394abf780-kube-api-access-j9qh5\") pod \"payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5\" (UID: \"86e79ca8-6f1f-421d-b8f8-343394abf780\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5" Nov 03 02:20:55.918728 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:55.918577 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86e79ca8-6f1f-421d-b8f8-343394abf780-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5\" (UID: \"86e79ca8-6f1f-421d-b8f8-343394abf780\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5" Nov 03 02:20:55.924237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:55.923877 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686"] Nov 03 02:20:56.020820 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:56.020748 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9qh5\" (UniqueName: \"kubernetes.io/projected/86e79ca8-6f1f-421d-b8f8-343394abf780-kube-api-access-j9qh5\") pod \"payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5\" (UID: \"86e79ca8-6f1f-421d-b8f8-343394abf780\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5" Nov 03 02:20:56.020820 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:56.020800 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86e79ca8-6f1f-421d-b8f8-343394abf780-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5\" (UID: \"86e79ca8-6f1f-421d-b8f8-343394abf780\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5" Nov 03 02:20:56.020982 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:56.020837 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3837cc6f-67ae-413a-ad00-3f2782fc8c5e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686\" (UID: \"3837cc6f-67ae-413a-ad00-3f2782fc8c5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686" Nov 03 02:20:56.020982 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:56.020945 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk4m9\" (UniqueName: \"kubernetes.io/projected/3837cc6f-67ae-413a-ad00-3f2782fc8c5e-kube-api-access-gk4m9\") pod \"payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686\" (UID: \"3837cc6f-67ae-413a-ad00-3f2782fc8c5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686" Nov 03 02:20:56.021531 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:56.021512 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86e79ca8-6f1f-421d-b8f8-343394abf780-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5\" (UID: \"86e79ca8-6f1f-421d-b8f8-343394abf780\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5" Nov 03 02:20:56.077734 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:56.077697 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9qh5\" (UniqueName: \"kubernetes.io/projected/86e79ca8-6f1f-421d-b8f8-343394abf780-kube-api-access-j9qh5\") pod \"payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5\" (UID: \"86e79ca8-6f1f-421d-b8f8-343394abf780\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5" Nov 03 02:20:56.122166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:56.122134 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk4m9\" (UniqueName: \"kubernetes.io/projected/3837cc6f-67ae-413a-ad00-3f2782fc8c5e-kube-api-access-gk4m9\") pod \"payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686\" (UID: \"3837cc6f-67ae-413a-ad00-3f2782fc8c5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686" Nov 03 02:20:56.122380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:56.122285 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3837cc6f-67ae-413a-ad00-3f2782fc8c5e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686\" (UID: \"3837cc6f-67ae-413a-ad00-3f2782fc8c5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686" Nov 03 02:20:56.122926 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:56.122906 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3837cc6f-67ae-413a-ad00-3f2782fc8c5e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686\" (UID: \"3837cc6f-67ae-413a-ad00-3f2782fc8c5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686" Nov 03 02:20:56.178699 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:56.178674 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk4m9\" (UniqueName: \"kubernetes.io/projected/3837cc6f-67ae-413a-ad00-3f2782fc8c5e-kube-api-access-gk4m9\") pod \"payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686\" (UID: \"3837cc6f-67ae-413a-ad00-3f2782fc8c5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686" Nov 03 02:20:56.235407 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:56.235376 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686" Nov 03 02:20:56.291982 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:56.291946 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5" Nov 03 02:20:56.606137 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:20:56.606103 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3837cc6f_67ae_413a_ad00_3f2782fc8c5e.slice/crio-40017cb8c11f383b35d4b636f5f4209f330fcb10d121062d8a9f7687b1d24093 WatchSource:0}: Error finding container 40017cb8c11f383b35d4b636f5f4209f330fcb10d121062d8a9f7687b1d24093: Status 404 returned error can't find the container with id 40017cb8c11f383b35d4b636f5f4209f330fcb10d121062d8a9f7687b1d24093 Nov 03 02:20:56.609688 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:56.609663 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686"] Nov 03 02:20:56.702143 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:20:56.702115 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e79ca8_6f1f_421d_b8f8_343394abf780.slice/crio-ccab718d8ec2d3eb5fa57d3fc8517aa6df28dae76a0fd448c07f9641b6ad99ca WatchSource:0}: Error finding container ccab718d8ec2d3eb5fa57d3fc8517aa6df28dae76a0fd448c07f9641b6ad99ca: Status 404 returned error can't find the container with id ccab718d8ec2d3eb5fa57d3fc8517aa6df28dae76a0fd448c07f9641b6ad99ca Nov 03 02:20:56.704977 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:56.704953 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5"] Nov 03 02:20:57.161908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:57.161862 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5" event={"ID":"86e79ca8-6f1f-421d-b8f8-343394abf780","Type":"ContainerStarted","Data":"b19d15b504fcdb913c5eb9326e5bd544ecd0f18d0ca8dd18f05ffc5ee6162d9d"} Nov 03 02:20:57.162075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:57.161936 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5" event={"ID":"86e79ca8-6f1f-421d-b8f8-343394abf780","Type":"ContainerStarted","Data":"ccab718d8ec2d3eb5fa57d3fc8517aa6df28dae76a0fd448c07f9641b6ad99ca"} Nov 03 02:20:57.163154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:57.163129 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5" Nov 03 02:20:57.165606 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:57.165574 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686" event={"ID":"3837cc6f-67ae-413a-ad00-3f2782fc8c5e","Type":"ContainerStarted","Data":"d4ddeee50937b125e831958c26c56b246558697490bcba17287666725a468666"} Nov 03 02:20:57.165692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:57.165634 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686" event={"ID":"3837cc6f-67ae-413a-ad00-3f2782fc8c5e","Type":"ContainerStarted","Data":"40017cb8c11f383b35d4b636f5f4209f330fcb10d121062d8a9f7687b1d24093"} Nov 03 02:20:57.166481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:57.166460 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686" Nov 03 02:20:57.276835 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:57.276772 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5" podStartSLOduration=2.276753173 podStartE2EDuration="2.276753173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:20:57.275645611 +0000 UTC m=+1738.959754765" watchObservedRunningTime="2026-11-03 02:20:57.276753173 +0000 UTC m=+1738.960862329" Nov 03 02:20:57.404485 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:57.395034 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686" podStartSLOduration=2.395012703 podStartE2EDuration="2.395012703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:20:57.391114642 +0000 UTC m=+1739.075223798" watchObservedRunningTime="2026-11-03 02:20:57.395012703 +0000 UTC m=+1739.079121861" Nov 03 02:20:59.031801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.031776 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:20:59.074292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.074116 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:59.087892 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.087761 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-workload-socket\") pod \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " Nov 03 02:20:59.088043 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.087908 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istiod-ca-cert\") pod \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " Nov 03 02:20:59.088043 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.087963 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-kube-api-access-vzw97\" (UniqueName: \"kubernetes.io/projected/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-kube-api-access-vzw97\") pod \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " Nov 03 02:20:59.088043 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.088004 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-workload-certs\") pod \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " Nov 03 02:20:59.088400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.088050 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-podinfo\") pod \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " Nov 03 02:20:59.088400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.088095 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-workload-socket" pod "9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" (UID: "9c7ab4d3-51b0-465b-9b0e-9fb811570ffb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:59.088400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.088114 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-envoy\") pod \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " Nov 03 02:20:59.088400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.088157 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-data\") pod \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " Nov 03 02:20:59.088400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.088223 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-token\" (UniqueName: \"kubernetes.io/projected/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-token\") pod \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " Nov 03 02:20:59.088400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.088271 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf" Nov 03 02:20:59.088400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.088285 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-credential-socket\") pod \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\" (UID: \"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb\") " Nov 03 02:20:59.089522 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.088928 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:59.089522 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.088971 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk" Nov 03 02:20:59.089522 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.089278 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-data" pod "9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" (UID: "9c7ab4d3-51b0-465b-9b0e-9fb811570ffb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:59.089522 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.089454 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-workload-certs" pod "9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" (UID: "9c7ab4d3-51b0-465b-9b0e-9fb811570ffb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:59.089788 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.089520 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istiod-ca-cert" pod "9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" (UID: "9c7ab4d3-51b0-465b-9b0e-9fb811570ffb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:20:59.090534 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.090045 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-credential-socket" pod "9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" (UID: "9c7ab4d3-51b0-465b-9b0e-9fb811570ffb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:59.091052 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.091018 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-podinfo" pod "9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" (UID: "9c7ab4d3-51b0-465b-9b0e-9fb811570ffb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:20:59.091724 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.091696 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-envoy" pod "9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" (UID: "9c7ab4d3-51b0-465b-9b0e-9fb811570ffb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:59.091724 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.091712 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-token" pod "9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" (UID: "9c7ab4d3-51b0-465b-9b0e-9fb811570ffb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:20:59.091858 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.091698 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-kube-api-access-vzw97" pod "9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" (UID: "9c7ab4d3-51b0-465b-9b0e-9fb811570ffb"). InnerVolumeSpecName "kube-api-access-vzw97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:20:59.107235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.107217 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:20:59.183667 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.183636 5463 generic.go:356] "Generic (PLEG): container finished" podID="5673c3cf-3606-42b2-968b-8ccec7e0233a" containerID="34bc20f65d467ba0d8ac2f5ca61e960260362053807362858bfbeffb8f54c51f" exitCode=0 Nov 03 02:20:59.183824 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.183686 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" Nov 03 02:20:59.183824 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.183721 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" event={"ID":"5673c3cf-3606-42b2-968b-8ccec7e0233a","Type":"ContainerDied","Data":"34bc20f65d467ba0d8ac2f5ca61e960260362053807362858bfbeffb8f54c51f"} Nov 03 02:20:59.183824 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.183772 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46" event={"ID":"5673c3cf-3606-42b2-968b-8ccec7e0233a","Type":"ContainerDied","Data":"185f0a1ee841262816480944d1105454a19d62b4830f0e5bec223b34312b95be"} Nov 03 02:20:59.183824 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.183801 5463 scope.go:122] "RemoveContainer" containerID="34bc20f65d467ba0d8ac2f5ca61e960260362053807362858bfbeffb8f54c51f" Nov 03 02:20:59.185416 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.185390 5463 generic.go:356] "Generic (PLEG): container finished" podID="9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" containerID="4503c097403c59562ca7ae1334e8a7617af7cfe4f49c02a73c63404faedcfffc" exitCode=0 Nov 03 02:20:59.185553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.185443 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" Nov 03 02:20:59.185553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.185474 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" event={"ID":"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb","Type":"ContainerDied","Data":"4503c097403c59562ca7ae1334e8a7617af7cfe4f49c02a73c63404faedcfffc"} Nov 03 02:20:59.185553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.185542 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc" event={"ID":"9c7ab4d3-51b0-465b-9b0e-9fb811570ffb","Type":"ContainerDied","Data":"77157e4ffc0865b110b39afd465dc79032f516815369b86932167e4697e414aa"} Nov 03 02:20:59.189293 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.189266 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-token\" (UniqueName: \"kubernetes.io/projected/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-token\") pod \"5673c3cf-3606-42b2-968b-8ccec7e0233a\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " Nov 03 02:20:59.189715 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.189680 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-workload-socket" pod "5673c3cf-3606-42b2-968b-8ccec7e0233a" (UID: "5673c3cf-3606-42b2-968b-8ccec7e0233a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:59.189715 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.189331 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-workload-socket\") pod \"5673c3cf-3606-42b2-968b-8ccec7e0233a\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " Nov 03 02:20:59.189912 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.189764 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5673c3cf-3606-42b2-968b-8ccec7e0233a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5673c3cf-3606-42b2-968b-8ccec7e0233a-istiod-ca-cert\") pod \"5673c3cf-3606-42b2-968b-8ccec7e0233a\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " Nov 03 02:20:59.189912 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.189821 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-podinfo\") pod \"5673c3cf-3606-42b2-968b-8ccec7e0233a\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " Nov 03 02:20:59.190062 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.189941 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-data\") pod \"5673c3cf-3606-42b2-968b-8ccec7e0233a\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " Nov 03 02:20:59.190062 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.190013 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-credential-socket\") pod \"5673c3cf-3606-42b2-968b-8ccec7e0233a\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " Nov 03 02:20:59.190173 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.190066 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-envoy\") pod \"5673c3cf-3606-42b2-968b-8ccec7e0233a\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " Nov 03 02:20:59.190173 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.190163 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5673c3cf-3606-42b2-968b-8ccec7e0233a-kube-api-access-gr2d9\" (UniqueName: \"kubernetes.io/projected/5673c3cf-3606-42b2-968b-8ccec7e0233a-kube-api-access-gr2d9\") pod \"5673c3cf-3606-42b2-968b-8ccec7e0233a\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " Nov 03 02:20:59.190385 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.190275 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-workload-certs\") pod \"5673c3cf-3606-42b2-968b-8ccec7e0233a\" (UID: \"5673c3cf-3606-42b2-968b-8ccec7e0233a\") " Nov 03 02:20:59.190385 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.190280 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5673c3cf-3606-42b2-968b-8ccec7e0233a-istiod-ca-cert" pod "5673c3cf-3606-42b2-968b-8ccec7e0233a" (UID: "5673c3cf-3606-42b2-968b-8ccec7e0233a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:20:59.190557 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.190532 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-credential-socket" pod "5673c3cf-3606-42b2-968b-8ccec7e0233a" (UID: "5673c3cf-3606-42b2-968b-8ccec7e0233a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:59.190993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.190960 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:59.190993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.190987 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5673c3cf-3606-42b2-968b-8ccec7e0233a-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:59.191141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.191002 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:59.191141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.191019 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:59.191141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.191034 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzw97\" (UniqueName: \"kubernetes.io/projected/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-kube-api-access-vzw97\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:59.191141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.191048 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:59.191141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.191062 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:59.191141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.191076 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:59.191141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.191091 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:59.191141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.191105 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:59.191141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.191119 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:59.191141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.190965 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-workload-certs" pod "5673c3cf-3606-42b2-968b-8ccec7e0233a" (UID: "5673c3cf-3606-42b2-968b-8ccec7e0233a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:59.191743 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.191140 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-data" pod "5673c3cf-3606-42b2-968b-8ccec7e0233a" (UID: "5673c3cf-3606-42b2-968b-8ccec7e0233a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:59.192907 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.192880 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-envoy" pod "5673c3cf-3606-42b2-968b-8ccec7e0233a" (UID: "5673c3cf-3606-42b2-968b-8ccec7e0233a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:20:59.193812 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.193616 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-token" pod "5673c3cf-3606-42b2-968b-8ccec7e0233a" (UID: "5673c3cf-3606-42b2-968b-8ccec7e0233a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:20:59.193812 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.193806 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5673c3cf-3606-42b2-968b-8ccec7e0233a-kube-api-access-gr2d9" pod "5673c3cf-3606-42b2-968b-8ccec7e0233a" (UID: "5673c3cf-3606-42b2-968b-8ccec7e0233a"). InnerVolumeSpecName "kube-api-access-gr2d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:20:59.196345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.194518 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-podinfo" pod "5673c3cf-3606-42b2-968b-8ccec7e0233a" (UID: "5673c3cf-3606-42b2-968b-8ccec7e0233a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:20:59.197229 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.197179 5463 scope.go:122] "RemoveContainer" containerID="34bc20f65d467ba0d8ac2f5ca61e960260362053807362858bfbeffb8f54c51f" Nov 03 02:20:59.197625 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:20:59.197515 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34bc20f65d467ba0d8ac2f5ca61e960260362053807362858bfbeffb8f54c51f\": container with ID starting with 34bc20f65d467ba0d8ac2f5ca61e960260362053807362858bfbeffb8f54c51f not found: ID does not exist" containerID="34bc20f65d467ba0d8ac2f5ca61e960260362053807362858bfbeffb8f54c51f" Nov 03 02:20:59.197625 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.197548 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34bc20f65d467ba0d8ac2f5ca61e960260362053807362858bfbeffb8f54c51f"} err="failed to get container status \"34bc20f65d467ba0d8ac2f5ca61e960260362053807362858bfbeffb8f54c51f\": rpc error: code = NotFound desc = could not find container \"34bc20f65d467ba0d8ac2f5ca61e960260362053807362858bfbeffb8f54c51f\": container with ID starting with 34bc20f65d467ba0d8ac2f5ca61e960260362053807362858bfbeffb8f54c51f not found: ID does not exist" Nov 03 02:20:59.197625 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.197571 5463 scope.go:122] "RemoveContainer" containerID="4503c097403c59562ca7ae1334e8a7617af7cfe4f49c02a73c63404faedcfffc" Nov 03 02:20:59.209140 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.209123 5463 scope.go:122] "RemoveContainer" containerID="4503c097403c59562ca7ae1334e8a7617af7cfe4f49c02a73c63404faedcfffc" Nov 03 02:20:59.209418 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:20:59.209400 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4503c097403c59562ca7ae1334e8a7617af7cfe4f49c02a73c63404faedcfffc\": container with ID starting with 4503c097403c59562ca7ae1334e8a7617af7cfe4f49c02a73c63404faedcfffc not found: ID does not exist" containerID="4503c097403c59562ca7ae1334e8a7617af7cfe4f49c02a73c63404faedcfffc" Nov 03 02:20:59.209471 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.209425 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4503c097403c59562ca7ae1334e8a7617af7cfe4f49c02a73c63404faedcfffc"} err="failed to get container status \"4503c097403c59562ca7ae1334e8a7617af7cfe4f49c02a73c63404faedcfffc\": rpc error: code = NotFound desc = could not find container \"4503c097403c59562ca7ae1334e8a7617af7cfe4f49c02a73c63404faedcfffc\": container with ID starting with 4503c097403c59562ca7ae1334e8a7617af7cfe4f49c02a73c63404faedcfffc not found: ID does not exist" Nov 03 02:20:59.292769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.292687 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:59.292769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.292725 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr2d9\" (UniqueName: \"kubernetes.io/projected/5673c3cf-3606-42b2-968b-8ccec7e0233a-kube-api-access-gr2d9\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:59.292769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.292743 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:59.292769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.292758 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:59.292999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.292773 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:59.292999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.292788 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5673c3cf-3606-42b2-968b-8ccec7e0233a-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:59.578351 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.578258 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc"] Nov 03 02:20:59.579880 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.579850 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-gmsxc"] Nov 03 02:20:59.830454 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.830368 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46"] Nov 03 02:20:59.833394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:20:59.833370 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-j4m46"] Nov 03 02:21:00.373711 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:00.373681 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:00.443810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:00.443781 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5673c3cf-3606-42b2-968b-8ccec7e0233a" path="/var/lib/kubelet/pods/5673c3cf-3606-42b2-968b-8ccec7e0233a/volumes" Nov 03 02:21:00.444275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:00.444260 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" path="/var/lib/kubelet/pods/9c7ab4d3-51b0-465b-9b0e-9fb811570ffb/volumes" Nov 03 02:21:00.529187 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:00.529156 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:00.709646 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:00.709612 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:04.532846 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:04.532754 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p"] Nov 03 02:21:04.533320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:04.532991 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" podUID="f1a2b779-6480-46dc-b9d2-05c3ef91bcb7" containerName="maas-api" containerID="cri-o://8942618ae8b97f547144c5fe9900ec6b7879f84a1942e026cca2a949ab2ed405" gracePeriod=30 Nov 03 02:21:04.654339 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:04.654298 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:04.851668 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:04.851642 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" Nov 03 02:21:04.999971 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:04.999936 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-kube-api-access-79qdg\" (UniqueName: \"kubernetes.io/projected/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-kube-api-access-79qdg\") pod \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\" (UID: \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\") " Nov 03 02:21:05.000170 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:04.999980 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-metrics-certs\") pod \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\" (UID: \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\") " Nov 03 02:21:05.000170 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:05.000012 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-openshift-service-ca\") pod \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\" (UID: \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\") " Nov 03 02:21:05.000170 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:05.000064 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-maas-api-tls\") pod \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\" (UID: \"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7\") " Nov 03 02:21:05.001246 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:05.000516 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-openshift-service-ca" pod "f1a2b779-6480-46dc-b9d2-05c3ef91bcb7" (UID: "f1a2b779-6480-46dc-b9d2-05c3ef91bcb7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:21:05.001246 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:05.000827 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:21:05.002922 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:05.002895 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-metrics-certs" pod "f1a2b779-6480-46dc-b9d2-05c3ef91bcb7" (UID: "f1a2b779-6480-46dc-b9d2-05c3ef91bcb7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:21:05.003048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:05.002921 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-kube-api-access-79qdg" pod "f1a2b779-6480-46dc-b9d2-05c3ef91bcb7" (UID: "f1a2b779-6480-46dc-b9d2-05c3ef91bcb7"). InnerVolumeSpecName "kube-api-access-79qdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:21:05.025668 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:05.025642 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-maas-api-tls" pod "f1a2b779-6480-46dc-b9d2-05c3ef91bcb7" (UID: "f1a2b779-6480-46dc-b9d2-05c3ef91bcb7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:21:05.104481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:05.104384 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79qdg\" (UniqueName: \"kubernetes.io/projected/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-kube-api-access-79qdg\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:21:05.104481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:05.104430 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:21:05.104481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:05.104446 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:21:05.238377 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:05.238337 5463 generic.go:356] "Generic (PLEG): container finished" podID="f1a2b779-6480-46dc-b9d2-05c3ef91bcb7" containerID="8942618ae8b97f547144c5fe9900ec6b7879f84a1942e026cca2a949ab2ed405" exitCode=0 Nov 03 02:21:05.238578 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:05.238390 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" Nov 03 02:21:05.238578 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:05.238423 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" event={"ID":"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7","Type":"ContainerDied","Data":"8942618ae8b97f547144c5fe9900ec6b7879f84a1942e026cca2a949ab2ed405"} Nov 03 02:21:05.238578 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:05.238492 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p" event={"ID":"f1a2b779-6480-46dc-b9d2-05c3ef91bcb7","Type":"ContainerDied","Data":"0bd9ba7da4df16493704e0bcdfc44d8bcc63343fca12314a313a6a7baf30ad6e"} Nov 03 02:21:05.238578 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:05.238514 5463 scope.go:122] "RemoveContainer" containerID="8942618ae8b97f547144c5fe9900ec6b7879f84a1942e026cca2a949ab2ed405" Nov 03 02:21:05.251895 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:05.251872 5463 scope.go:122] "RemoveContainer" containerID="8942618ae8b97f547144c5fe9900ec6b7879f84a1942e026cca2a949ab2ed405" Nov 03 02:21:05.252280 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:21:05.252249 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8942618ae8b97f547144c5fe9900ec6b7879f84a1942e026cca2a949ab2ed405\": container with ID starting with 8942618ae8b97f547144c5fe9900ec6b7879f84a1942e026cca2a949ab2ed405 not found: ID does not exist" containerID="8942618ae8b97f547144c5fe9900ec6b7879f84a1942e026cca2a949ab2ed405" Nov 03 02:21:05.252413 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:05.252288 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8942618ae8b97f547144c5fe9900ec6b7879f84a1942e026cca2a949ab2ed405"} err="failed to get container status \"8942618ae8b97f547144c5fe9900ec6b7879f84a1942e026cca2a949ab2ed405\": rpc error: code = NotFound desc = could not find container \"8942618ae8b97f547144c5fe9900ec6b7879f84a1942e026cca2a949ab2ed405\": container with ID starting with 8942618ae8b97f547144c5fe9900ec6b7879f84a1942e026cca2a949ab2ed405 not found: ID does not exist" Nov 03 02:21:05.404769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:05.404730 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p"] Nov 03 02:21:05.451719 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:05.451676 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9b509-7d565bc76-sx45p"] Nov 03 02:21:06.425114 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:06.425081 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:06.445633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:06.445597 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1a2b779-6480-46dc-b9d2-05c3ef91bcb7" path="/var/lib/kubelet/pods/f1a2b779-6480-46dc-b9d2-05c3ef91bcb7/volumes" Nov 03 02:21:08.074768 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:08.074667 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:08.233645 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:08.233604 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:08.262654 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:08.262626 5463 generic.go:356] "Generic (PLEG): container finished" podID="ba02d63a-95b8-4195-b07d-4f0e68398761" containerID="66b1b195d206e7e6365a1b490e570eba02f228ca6191131438bfde3c3e70e576" exitCode=0 Nov 03 02:21:08.262807 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:08.262704 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" event={"ID":"ba02d63a-95b8-4195-b07d-4f0e68398761","Type":"ContainerDied","Data":"66b1b195d206e7e6365a1b490e570eba02f228ca6191131438bfde3c3e70e576"} Nov 03 02:21:08.263132 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:08.263118 5463 scope.go:122] "RemoveContainer" containerID="66b1b195d206e7e6365a1b490e570eba02f228ca6191131438bfde3c3e70e576" Nov 03 02:21:08.264838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:08.264803 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:21:08.264911 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:08.264899 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:21:08.264959 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:08.264944 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:21:09.189001 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:09.188959 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5" Nov 03 02:21:09.189953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:09.189932 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686" Nov 03 02:21:09.198962 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:09.198932 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:09.286216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:09.285175 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" event={"ID":"ba02d63a-95b8-4195-b07d-4f0e68398761","Type":"ContainerStarted","Data":"faace85c23203886a42f5d8b71ea3f476a418b0f9b99794aefde5e320446646a"} Nov 03 02:21:09.439692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:09.439614 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:21:09.439838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:09.439782 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 02:21:09.439881 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:09.439831 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" podUID="ba02d63a-95b8-4195-b07d-4f0e68398761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 02:21:09.702567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:09.702437 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:10.237768 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:10.237736 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9"] Nov 03 02:21:10.238315 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:10.237965 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" podUID="84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" containerName="istio-proxy" containerID="cri-o://1893449848da4a7f7aa9a010f348b47cff64039d2c6a76535194214c3e0ed51b" gracePeriod=30 Nov 03 02:21:10.260718 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:10.260629 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:10.441067 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:10.441031 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 02:21:10.441268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:10.441113 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" podUID="ba02d63a-95b8-4195-b07d-4f0e68398761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 02:21:11.300761 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:11.300731 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:11.439656 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:11.439621 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 02:21:11.439844 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:11.439686 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" podUID="ba02d63a-95b8-4195-b07d-4f0e68398761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 02:21:11.470130 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:11.470089 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:11.987109 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:11.987071 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs"] Nov 03 02:21:11.988421 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:11.988396 5463 cpu_manager.go:414] "containerMap: removing container" podUID="f1a2b779-6480-46dc-b9d2-05c3ef91bcb7" containerName="maas-api" Nov 03 02:21:11.988570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:11.988558 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a2b779-6480-46dc-b9d2-05c3ef91bcb7" containerName="maas-api" Nov 03 02:21:11.988737 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:11.988725 5463 cpu_manager.go:414] "containerMap: removing container" podUID="9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" containerName="istio-proxy" Nov 03 02:21:11.988817 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:11.988807 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" containerName="istio-proxy" Nov 03 02:21:11.988910 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:11.988901 5463 cpu_manager.go:414] "containerMap: removing container" podUID="5673c3cf-3606-42b2-968b-8ccec7e0233a" containerName="istio-proxy" Nov 03 02:21:11.988977 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:11.988969 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5673c3cf-3606-42b2-968b-8ccec7e0233a" containerName="istio-proxy" Nov 03 02:21:11.989212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:11.989181 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c7ab4d3-51b0-465b-9b0e-9fb811570ffb" containerName="istio-proxy" Nov 03 02:21:11.989315 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:11.989305 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="5673c3cf-3606-42b2-968b-8ccec7e0233a" containerName="istio-proxy" Nov 03 02:21:11.989391 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:11.989383 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a2b779-6480-46dc-b9d2-05c3ef91bcb7" containerName="maas-api" Nov 03 02:21:11.993119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:11.993101 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.022946 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.022915 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6smsm\" (UniqueName: \"kubernetes.io/projected/a7d39617-aba6-4951-b3ce-14f544abbf7a-kube-api-access-6smsm\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.023160 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.023138 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-envoy\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.023335 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.023314 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-workload-socket\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.023452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.023435 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-data\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.023520 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.023500 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-token\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.023576 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.023537 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-workload-certs\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.023652 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.023639 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d39617-aba6-4951-b3ce-14f544abbf7a-istiod-ca-cert\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.023716 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.023697 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-podinfo\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.023767 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.023726 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-credential-socket\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.040413 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.040098 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs"] Nov 03 02:21:12.099288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.099245 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-56299c-openshift-default-dockercfg-lm2x6\"" Nov 03 02:21:12.113262 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.113162 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:12.125133 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.125105 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-workload-socket\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.125296 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.125157 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-data\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.125296 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.125222 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-token\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.125296 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.125243 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-workload-certs\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.125406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.125296 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d39617-aba6-4951-b3ce-14f544abbf7a-istiod-ca-cert\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.125406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.125328 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-podinfo\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.125406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.125343 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-credential-socket\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.125406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.125390 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6smsm\" (UniqueName: \"kubernetes.io/projected/a7d39617-aba6-4951-b3ce-14f544abbf7a-kube-api-access-6smsm\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.125533 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.125421 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-envoy\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.125973 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.125950 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-workload-certs\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.126154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.125973 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-data\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.126154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.126037 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-workload-socket\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.126302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.126069 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-credential-socket\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.126775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.126531 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d39617-aba6-4951-b3ce-14f544abbf7a-istiod-ca-cert\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.128365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.128346 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-envoy\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.128501 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.128475 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-podinfo\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.203225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.203146 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-token\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.204406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.204270 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6smsm\" (UniqueName: \"kubernetes.io/projected/a7d39617-aba6-4951-b3ce-14f544abbf7a-kube-api-access-6smsm\") pod \"e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.290116 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.289995 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:12.311527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.311485 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:12.441461 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.441407 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 02:21:12.441616 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.441467 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" podUID="ba02d63a-95b8-4195-b07d-4f0e68398761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 02:21:12.778940 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:21:12.778342 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d39617_aba6_4951_b3ce_14f544abbf7a.slice/crio-8bd817560f5f8808e3a76853c06e6d62521ec5c2cfeb892de80030324e4b6deb WatchSource:0}: Error finding container 8bd817560f5f8808e3a76853c06e6d62521ec5c2cfeb892de80030324e4b6deb: Status 404 returned error can't find the container with id 8bd817560f5f8808e3a76853c06e6d62521ec5c2cfeb892de80030324e4b6deb Nov 03 02:21:12.781003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.780960 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:21:12.781093 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.781051 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:21:12.781133 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.781094 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:21:12.782625 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.782603 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs"] Nov 03 02:21:12.914559 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.914527 5463 patch_prober.go:29] interesting pod/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:21:12.914751 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.914578 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" podUID="84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:21:12.989341 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:12.989309 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:13.007064 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:21:13.007029 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:21:13.329167 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:13.329132 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" event={"ID":"a7d39617-aba6-4951-b3ce-14f544abbf7a","Type":"ContainerStarted","Data":"42e9941f147f6c7e61d605cdffa98cc6a0b7707baa76c6853888e3011bd2fa7e"} Nov 03 02:21:13.329779 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:13.329218 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" event={"ID":"a7d39617-aba6-4951-b3ce-14f544abbf7a","Type":"ContainerStarted","Data":"8bd817560f5f8808e3a76853c06e6d62521ec5c2cfeb892de80030324e4b6deb"} Nov 03 02:21:13.440342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:13.440312 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 02:21:13.440566 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:13.440358 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" podUID="ba02d63a-95b8-4195-b07d-4f0e68398761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 02:21:13.468684 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:13.468438 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" podStartSLOduration=2.46841814 podStartE2EDuration="2.46841814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:21:13.444871242 +0000 UTC m=+1755.128980397" watchObservedRunningTime="2026-11-03 02:21:13.46841814 +0000 UTC m=+1755.152527300" Nov 03 02:21:14.314264 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:14.314227 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:14.320203 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:14.320163 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:14.338374 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:14.338348 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:14.339114 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:14.339093 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:21:14.503100 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:14.503069 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:21:14.504107 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:14.504085 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:21:14.504935 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:14.504914 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:21:15.508480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.508414 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:21:15.587856 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.587824 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-kube-api-access-62z22\" (UniqueName: \"kubernetes.io/projected/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-kube-api-access-62z22\") pod \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " Nov 03 02:21:15.588008 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.587927 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istiod-ca-cert\") pod \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " Nov 03 02:21:15.588008 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.587976 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-credential-socket\") pod \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " Nov 03 02:21:15.588107 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.588028 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-podinfo\") pod \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " Nov 03 02:21:15.588107 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.588080 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-workload-certs\") pod \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " Nov 03 02:21:15.588225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.588139 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-data\") pod \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " Nov 03 02:21:15.588295 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.588238 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-token\" (UniqueName: \"kubernetes.io/projected/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-token\") pod \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " Nov 03 02:21:15.588358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.588318 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-credential-socket" pod "84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" (UID: "84137e59-ff98-4062-b0d7-7fa4f2fbdc9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:21:15.588421 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.588368 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-envoy\") pod \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " Nov 03 02:21:15.588481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.588407 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-workload-socket\") pod \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\" (UID: \"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e\") " Nov 03 02:21:15.588481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.588421 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istiod-ca-cert" pod "84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" (UID: "84137e59-ff98-4062-b0d7-7fa4f2fbdc9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:21:15.588598 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.588538 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-data" pod "84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" (UID: "84137e59-ff98-4062-b0d7-7fa4f2fbdc9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:21:15.588691 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.588669 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-workload-certs" pod "84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" (UID: "84137e59-ff98-4062-b0d7-7fa4f2fbdc9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:21:15.588780 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.588753 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-workload-socket" pod "84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" (UID: "84137e59-ff98-4062-b0d7-7fa4f2fbdc9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:21:15.589013 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.588993 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:21:15.589141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.589016 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:21:15.589141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.589033 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:21:15.589141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.589046 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:21:15.589141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.589059 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:21:15.590487 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.590461 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-podinfo" pod "84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" (UID: "84137e59-ff98-4062-b0d7-7fa4f2fbdc9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:21:15.590596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.590582 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-token" pod "84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" (UID: "84137e59-ff98-4062-b0d7-7fa4f2fbdc9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:21:15.590662 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.590634 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-kube-api-access-62z22" pod "84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" (UID: "84137e59-ff98-4062-b0d7-7fa4f2fbdc9e"). InnerVolumeSpecName "kube-api-access-62z22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:21:15.590824 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.590808 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-envoy" pod "84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" (UID: "84137e59-ff98-4062-b0d7-7fa4f2fbdc9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:21:15.690345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.690307 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:21:15.690345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.690343 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:21:15.690536 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.690359 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62z22\" (UniqueName: \"kubernetes.io/projected/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-kube-api-access-62z22\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:21:15.690536 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:15.690376 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:21:16.230390 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:16.230354 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:16.358702 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:16.358671 5463 generic.go:356] "Generic (PLEG): container finished" podID="84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" containerID="1893449848da4a7f7aa9a010f348b47cff64039d2c6a76535194214c3e0ed51b" exitCode=0 Nov 03 02:21:16.358879 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:16.358721 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" Nov 03 02:21:16.358879 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:16.358748 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" event={"ID":"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e","Type":"ContainerDied","Data":"1893449848da4a7f7aa9a010f348b47cff64039d2c6a76535194214c3e0ed51b"} Nov 03 02:21:16.358879 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:16.358797 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9" event={"ID":"84137e59-ff98-4062-b0d7-7fa4f2fbdc9e","Type":"ContainerDied","Data":"01882f8ba45b378bcf58ac613d04d91e4f18d8640b19842c0ce9d24c7bbee93b"} Nov 03 02:21:16.358879 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:16.358823 5463 scope.go:122] "RemoveContainer" containerID="1893449848da4a7f7aa9a010f348b47cff64039d2c6a76535194214c3e0ed51b" Nov 03 02:21:16.370675 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:16.370660 5463 scope.go:122] "RemoveContainer" containerID="e1667115830b7fe3fae41cf011b4b63a9805ba45cb461ee9652d11320dbda32c" Nov 03 02:21:16.380048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:16.380028 5463 scope.go:122] "RemoveContainer" containerID="1893449848da4a7f7aa9a010f348b47cff64039d2c6a76535194214c3e0ed51b" Nov 03 02:21:16.380430 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:21:16.380401 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1893449848da4a7f7aa9a010f348b47cff64039d2c6a76535194214c3e0ed51b\": container with ID starting with 1893449848da4a7f7aa9a010f348b47cff64039d2c6a76535194214c3e0ed51b not found: ID does not exist" containerID="1893449848da4a7f7aa9a010f348b47cff64039d2c6a76535194214c3e0ed51b" Nov 03 02:21:16.380527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:16.380440 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1893449848da4a7f7aa9a010f348b47cff64039d2c6a76535194214c3e0ed51b"} err="failed to get container status \"1893449848da4a7f7aa9a010f348b47cff64039d2c6a76535194214c3e0ed51b\": rpc error: code = NotFound desc = could not find container \"1893449848da4a7f7aa9a010f348b47cff64039d2c6a76535194214c3e0ed51b\": container with ID starting with 1893449848da4a7f7aa9a010f348b47cff64039d2c6a76535194214c3e0ed51b not found: ID does not exist" Nov 03 02:21:16.380527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:16.380463 5463 scope.go:122] "RemoveContainer" containerID="e1667115830b7fe3fae41cf011b4b63a9805ba45cb461ee9652d11320dbda32c" Nov 03 02:21:16.380748 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:21:16.380724 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1667115830b7fe3fae41cf011b4b63a9805ba45cb461ee9652d11320dbda32c\": container with ID starting with e1667115830b7fe3fae41cf011b4b63a9805ba45cb461ee9652d11320dbda32c not found: ID does not exist" containerID="e1667115830b7fe3fae41cf011b4b63a9805ba45cb461ee9652d11320dbda32c" Nov 03 02:21:16.380801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:16.380758 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1667115830b7fe3fae41cf011b4b63a9805ba45cb461ee9652d11320dbda32c"} err="failed to get container status \"e1667115830b7fe3fae41cf011b4b63a9805ba45cb461ee9652d11320dbda32c\": rpc error: code = NotFound desc = could not find container \"e1667115830b7fe3fae41cf011b4b63a9805ba45cb461ee9652d11320dbda32c\": container with ID starting with e1667115830b7fe3fae41cf011b4b63a9805ba45cb461ee9652d11320dbda32c not found: ID does not exist" Nov 03 02:21:16.423699 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:16.423673 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:16.578680 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:16.578595 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9"] Nov 03 02:21:16.580526 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:16.580496 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:16.626971 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:16.626937 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f9b509-openshift-default-9f6576965-j84n9"] Nov 03 02:21:18.444295 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:18.444261 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" path="/var/lib/kubelet/pods/84137e59-ff98-4062-b0d7-7fa4f2fbdc9e/volumes" Nov 03 02:21:19.487343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:19.487308 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:19.669870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:19.669837 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:19.888181 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:19.888092 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk"] Nov 03 02:21:19.888884 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:19.888865 5463 cpu_manager.go:414] "containerMap: removing container" podUID="84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" containerName="istio-proxy" Nov 03 02:21:19.888884 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:19.888884 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" containerName="istio-proxy" Nov 03 02:21:19.889068 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:19.888928 5463 cpu_manager.go:414] "containerMap: removing container" podUID="84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" containerName="istio-proxy" Nov 03 02:21:19.889068 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:19.888936 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" containerName="istio-proxy" Nov 03 02:21:19.889275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:19.889252 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" containerName="istio-proxy" Nov 03 02:21:19.889407 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:19.889298 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="84137e59-ff98-4062-b0d7-7fa4f2fbdc9e" containerName="istio-proxy" Nov 03 02:21:19.892070 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:19.892052 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk" Nov 03 02:21:19.944130 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:19.944099 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk"] Nov 03 02:21:19.990796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:19.990764 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-56299c\"" Nov 03 02:21:20.067719 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:20.067678 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7j2l\" (UniqueName: \"kubernetes.io/projected/ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa-kube-api-access-q7j2l\") pod \"payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk\" (UID: \"ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk" Nov 03 02:21:20.067927 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:20.067789 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk\" (UID: \"ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk" Nov 03 02:21:20.076284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:20.076242 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2"] Nov 03 02:21:20.090253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:20.090223 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2" Nov 03 02:21:20.111606 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:20.111578 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2"] Nov 03 02:21:20.144475 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:20.144447 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-56299c-dockercfg-444sg\"" Nov 03 02:21:20.169033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:20.168997 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7j2l\" (UniqueName: \"kubernetes.io/projected/ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa-kube-api-access-q7j2l\") pod \"payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk\" (UID: \"ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk" Nov 03 02:21:20.169177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:20.169092 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk\" (UID: \"ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk" Nov 03 02:21:20.169306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:20.169279 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs5c4\" (UniqueName: \"kubernetes.io/projected/0d84ec5e-7992-48e5-83ab-816302748e41-kube-api-access-zs5c4\") pod \"payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2\" (UID: \"0d84ec5e-7992-48e5-83ab-816302748e41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2" Nov 03 02:21:20.169372 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:20.169313 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d84ec5e-7992-48e5-83ab-816302748e41-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2\" (UID: \"0d84ec5e-7992-48e5-83ab-816302748e41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2" Nov 03 02:21:20.169998 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:20.169967 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk\" (UID: \"ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk" Nov 03 02:21:20.225958 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:20.225928 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7j2l\" (UniqueName: \"kubernetes.io/projected/ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa-kube-api-access-q7j2l\") pod \"payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk\" (UID: \"ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk" Nov 03 02:21:20.270770 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:20.270741 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs5c4\" (UniqueName: \"kubernetes.io/projected/0d84ec5e-7992-48e5-83ab-816302748e41-kube-api-access-zs5c4\") pod \"payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2\" (UID: \"0d84ec5e-7992-48e5-83ab-816302748e41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2" Nov 03 02:21:20.270894 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:20.270779 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d84ec5e-7992-48e5-83ab-816302748e41-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2\" (UID: \"0d84ec5e-7992-48e5-83ab-816302748e41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2" Nov 03 02:21:20.271504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:20.271484 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d84ec5e-7992-48e5-83ab-816302748e41-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2\" (UID: \"0d84ec5e-7992-48e5-83ab-816302748e41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2" Nov 03 02:21:20.327455 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:20.327427 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs5c4\" (UniqueName: \"kubernetes.io/projected/0d84ec5e-7992-48e5-83ab-816302748e41-kube-api-access-zs5c4\") pod \"payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2\" (UID: \"0d84ec5e-7992-48e5-83ab-816302748e41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2" Nov 03 02:21:20.430603 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:20.430526 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2" Nov 03 02:21:20.507157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:20.506421 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk" Nov 03 02:21:20.805564 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:21:20.805523 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d84ec5e_7992_48e5_83ab_816302748e41.slice/crio-e5b05eac985330d33ad2f32338eddfbf9dda83568720d7ab7fe47594bcbbd970 WatchSource:0}: Error finding container e5b05eac985330d33ad2f32338eddfbf9dda83568720d7ab7fe47594bcbbd970: Status 404 returned error can't find the container with id e5b05eac985330d33ad2f32338eddfbf9dda83568720d7ab7fe47594bcbbd970 Nov 03 02:21:20.806512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:20.806478 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2"] Nov 03 02:21:20.904274 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:21:20.904231 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1a04a5_b15b_4092_aaf3_4c3f7b908ffa.slice/crio-12f274e25a77b6306e5a1a7dc96a70593d736970a014a5f9a363ae581b39e522 WatchSource:0}: Error finding container 12f274e25a77b6306e5a1a7dc96a70593d736970a014a5f9a363ae581b39e522: Status 404 returned error can't find the container with id 12f274e25a77b6306e5a1a7dc96a70593d736970a014a5f9a363ae581b39e522 Nov 03 02:21:20.908368 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:20.908343 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk"] Nov 03 02:21:21.401443 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:21.401407 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2" event={"ID":"0d84ec5e-7992-48e5-83ab-816302748e41","Type":"ContainerStarted","Data":"2abeb84d7418a307cf4751d8e2dcf3a956e17d3e878c9f404282d0347837074c"} Nov 03 02:21:21.401623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:21.401477 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2" Nov 03 02:21:21.401623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:21.401502 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2" event={"ID":"0d84ec5e-7992-48e5-83ab-816302748e41","Type":"ContainerStarted","Data":"e5b05eac985330d33ad2f32338eddfbf9dda83568720d7ab7fe47594bcbbd970"} Nov 03 02:21:21.403088 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:21.403065 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk" event={"ID":"ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa","Type":"ContainerStarted","Data":"603b32a12196d3e59b2fde25663b2b1efa65b727fd9a3bb3a4051e04b5d682ff"} Nov 03 02:21:21.403253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:21.403114 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk" event={"ID":"ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa","Type":"ContainerStarted","Data":"12f274e25a77b6306e5a1a7dc96a70593d736970a014a5f9a363ae581b39e522"} Nov 03 02:21:21.403253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:21.403166 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk" Nov 03 02:21:21.517710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:21.517663 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2" podStartSLOduration=1.517649955 podStartE2EDuration="1.517649955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:21:21.514343958 +0000 UTC m=+1763.198453114" watchObservedRunningTime="2026-11-03 02:21:21.517649955 +0000 UTC m=+1763.201759110" Nov 03 02:21:21.630761 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:21.630703 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk" podStartSLOduration=2.6306900730000002 podStartE2EDuration="2.630690073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:21:21.627787792 +0000 UTC m=+1763.311896949" watchObservedRunningTime="2026-11-03 02:21:21.630690073 +0000 UTC m=+1763.314799227" Nov 03 02:21:21.694770 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:21.694674 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:25.467663 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:25.467624 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:25.974095 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:25.974060 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:27.060402 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.060367 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9"] Nov 03 02:21:27.065156 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.065133 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9" Nov 03 02:21:27.116841 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.116805 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9"] Nov 03 02:21:27.162891 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.162861 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f66575\"" Nov 03 02:21:27.163047 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.162983 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f66575-dockercfg-k6gbj\"" Nov 03 02:21:27.164275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.164254 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7ebfc5-bca8-4c89-8dfc-fc486030f56f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9\" (UID: \"2d7ebfc5-bca8-4c89-8dfc-fc486030f56f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9" Nov 03 02:21:27.164404 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.164383 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d95cf\" (UniqueName: \"kubernetes.io/projected/2d7ebfc5-bca8-4c89-8dfc-fc486030f56f-kube-api-access-d95cf\") pod \"payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9\" (UID: \"2d7ebfc5-bca8-4c89-8dfc-fc486030f56f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9" Nov 03 02:21:27.266359 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.266321 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d95cf\" (UniqueName: \"kubernetes.io/projected/2d7ebfc5-bca8-4c89-8dfc-fc486030f56f-kube-api-access-d95cf\") pod \"payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9\" (UID: \"2d7ebfc5-bca8-4c89-8dfc-fc486030f56f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9" Nov 03 02:21:27.266763 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.266738 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7ebfc5-bca8-4c89-8dfc-fc486030f56f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9\" (UID: \"2d7ebfc5-bca8-4c89-8dfc-fc486030f56f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9" Nov 03 02:21:27.268149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.268125 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7ebfc5-bca8-4c89-8dfc-fc486030f56f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9\" (UID: \"2d7ebfc5-bca8-4c89-8dfc-fc486030f56f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9" Nov 03 02:21:27.326264 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.326165 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d95cf\" (UniqueName: \"kubernetes.io/projected/2d7ebfc5-bca8-4c89-8dfc-fc486030f56f-kube-api-access-d95cf\") pod \"payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9\" (UID: \"2d7ebfc5-bca8-4c89-8dfc-fc486030f56f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9" Nov 03 02:21:27.380484 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.380110 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9" Nov 03 02:21:27.776091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.776058 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9"] Nov 03 02:21:27.776784 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:21:27.776757 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7ebfc5_bca8_4c89_8dfc_fc486030f56f.slice/crio-896b627a63d9d9994245738b983da82163ae84f69c1ca3fe72e603d80f07edfc WatchSource:0}: Error finding container 896b627a63d9d9994245738b983da82163ae84f69c1ca3fe72e603d80f07edfc: Status 404 returned error can't find the container with id 896b627a63d9d9994245738b983da82163ae84f69c1ca3fe72e603d80f07edfc Nov 03 02:21:27.837133 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.837106 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48"] Nov 03 02:21:27.841231 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.841212 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48" Nov 03 02:21:27.880418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.880171 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvvvb\" (UniqueName: \"kubernetes.io/projected/8c13ed61-b2e6-40af-9e66-b9538243eaf7-kube-api-access-kvvvb\") pod \"payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48\" (UID: \"8c13ed61-b2e6-40af-9e66-b9538243eaf7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48" Nov 03 02:21:27.880418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.880296 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c13ed61-b2e6-40af-9e66-b9538243eaf7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48\" (UID: \"8c13ed61-b2e6-40af-9e66-b9538243eaf7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48" Nov 03 02:21:27.895385 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.895358 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48"] Nov 03 02:21:27.980976 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.980932 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvvvb\" (UniqueName: \"kubernetes.io/projected/8c13ed61-b2e6-40af-9e66-b9538243eaf7-kube-api-access-kvvvb\") pod \"payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48\" (UID: \"8c13ed61-b2e6-40af-9e66-b9538243eaf7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48" Nov 03 02:21:27.981178 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.981026 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c13ed61-b2e6-40af-9e66-b9538243eaf7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48\" (UID: \"8c13ed61-b2e6-40af-9e66-b9538243eaf7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48" Nov 03 02:21:27.981819 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:27.981797 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c13ed61-b2e6-40af-9e66-b9538243eaf7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48\" (UID: \"8c13ed61-b2e6-40af-9e66-b9538243eaf7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48" Nov 03 02:21:28.038550 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:28.038486 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvvvb\" (UniqueName: \"kubernetes.io/projected/8c13ed61-b2e6-40af-9e66-b9538243eaf7-kube-api-access-kvvvb\") pod \"payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48\" (UID: \"8c13ed61-b2e6-40af-9e66-b9538243eaf7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48" Nov 03 02:21:28.185325 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:28.185288 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48" Nov 03 02:21:28.493780 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:28.493712 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9" event={"ID":"2d7ebfc5-bca8-4c89-8dfc-fc486030f56f","Type":"ContainerStarted","Data":"f19e588bd0df73365707efddc1282b7e20e56f733c269f820fbab328bb9cf7fa"} Nov 03 02:21:28.493780 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:28.493770 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9" event={"ID":"2d7ebfc5-bca8-4c89-8dfc-fc486030f56f","Type":"ContainerStarted","Data":"896b627a63d9d9994245738b983da82163ae84f69c1ca3fe72e603d80f07edfc"} Nov 03 02:21:28.494442 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:28.493856 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9" Nov 03 02:21:28.602181 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:28.602159 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48"] Nov 03 02:21:28.618592 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:28.618547 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9" podStartSLOduration=1.618534482 podStartE2EDuration="1.618534482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:21:28.606796142 +0000 UTC m=+1770.290905296" watchObservedRunningTime="2026-11-03 02:21:28.618534482 +0000 UTC m=+1770.302643637" Nov 03 02:21:28.830246 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:28.830146 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:29.504866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:29.504829 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48" event={"ID":"8c13ed61-b2e6-40af-9e66-b9538243eaf7","Type":"ContainerStarted","Data":"d3851d400aa558020288b9a13d03b8d290ff019e258cbbdd2014ba06ceb4cf43"} Nov 03 02:21:29.505384 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:29.504891 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48" event={"ID":"8c13ed61-b2e6-40af-9e66-b9538243eaf7","Type":"ContainerStarted","Data":"7de9fcb50d6e3a5d3b3ac2f98c28601e62b7ed9f74968f7b5b0c731ed0d87ac9"} Nov 03 02:21:29.505384 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:29.505065 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48" Nov 03 02:21:29.622549 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:29.622498 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48" podStartSLOduration=2.622481307 podStartE2EDuration="2.622481307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:21:29.622128734 +0000 UTC m=+1771.306237889" watchObservedRunningTime="2026-11-03 02:21:29.622481307 +0000 UTC m=+1771.306590463" Nov 03 02:21:32.413400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:32.413365 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2" Nov 03 02:21:32.413779 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:32.413443 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk" Nov 03 02:21:35.396423 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:35.396350 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:36.571266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:36.571148 5463 generic.go:356] "Generic (PLEG): container finished" podID="871baea4-93af-43d2-8ef7-0b05c4f50526" containerID="58896957410a6ed50693266f6766b0bf5a3019aa55b56c1fbc1a520a40c4effb" exitCode=0 Nov 03 02:21:36.571266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:36.571227 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" event={"ID":"871baea4-93af-43d2-8ef7-0b05c4f50526","Type":"ContainerDied","Data":"58896957410a6ed50693266f6766b0bf5a3019aa55b56c1fbc1a520a40c4effb"} Nov 03 02:21:36.571733 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:36.571624 5463 scope.go:122] "RemoveContainer" containerID="58896957410a6ed50693266f6766b0bf5a3019aa55b56c1fbc1a520a40c4effb" Nov 03 02:21:36.573156 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:36.573125 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:21:36.573287 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:36.573207 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:21:36.573287 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:36.573243 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:21:37.586608 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:37.586575 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" event={"ID":"871baea4-93af-43d2-8ef7-0b05c4f50526","Type":"ContainerStarted","Data":"32c3825f11d1e75f6c0c9e054a0fffb4805429db3f9434bd87b75794ab07c0f3"} Nov 03 02:21:37.600443 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:37.600420 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:21:37.600625 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:37.600606 5463 patch_prober.go:29] interesting pod/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 02:21:37.600671 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:37.600649 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" podUID="871baea4-93af-43d2-8ef7-0b05c4f50526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 02:21:38.600860 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:38.600828 5463 patch_prober.go:29] interesting pod/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 02:21:38.601282 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:38.600873 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" podUID="871baea4-93af-43d2-8ef7-0b05c4f50526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 02:21:38.878369 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:38.878270 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:39.351385 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:39.350807 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:39.507795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:39.507761 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9" Nov 03 02:21:39.602696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:39.602591 5463 patch_prober.go:29] interesting pod/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 02:21:39.602696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:39.602663 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" podUID="871baea4-93af-43d2-8ef7-0b05c4f50526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 02:21:40.431185 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:40.431152 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr"] Nov 03 02:21:40.434522 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:40.434504 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr" Nov 03 02:21:40.487394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:40.487350 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr"] Nov 03 02:21:40.513366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:40.513336 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48" Nov 03 02:21:40.533079 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:40.533055 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b3bbbc-dockercfg-qzqnd\"" Nov 03 02:21:40.533079 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:40.533060 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b3bbbc\"" Nov 03 02:21:40.587682 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:40.587647 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33f6aaf-7588-488b-8b31-9ba405917c08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr\" (UID: \"a33f6aaf-7588-488b-8b31-9ba405917c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr" Nov 03 02:21:40.587891 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:40.587703 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z59dl\" (UniqueName: \"kubernetes.io/projected/a33f6aaf-7588-488b-8b31-9ba405917c08-kube-api-access-z59dl\") pod \"payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr\" (UID: \"a33f6aaf-7588-488b-8b31-9ba405917c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr" Nov 03 02:21:40.600882 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:40.600847 5463 patch_prober.go:29] interesting pod/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 02:21:40.601037 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:40.600893 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" podUID="871baea4-93af-43d2-8ef7-0b05c4f50526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 02:21:40.688814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:40.688726 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33f6aaf-7588-488b-8b31-9ba405917c08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr\" (UID: \"a33f6aaf-7588-488b-8b31-9ba405917c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr" Nov 03 02:21:40.688814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:40.688793 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z59dl\" (UniqueName: \"kubernetes.io/projected/a33f6aaf-7588-488b-8b31-9ba405917c08-kube-api-access-z59dl\") pod \"payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr\" (UID: \"a33f6aaf-7588-488b-8b31-9ba405917c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr" Nov 03 02:21:40.689662 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:40.689636 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33f6aaf-7588-488b-8b31-9ba405917c08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr\" (UID: \"a33f6aaf-7588-488b-8b31-9ba405917c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr" Nov 03 02:21:40.745757 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:40.745726 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z59dl\" (UniqueName: \"kubernetes.io/projected/a33f6aaf-7588-488b-8b31-9ba405917c08-kube-api-access-z59dl\") pod \"payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr\" (UID: \"a33f6aaf-7588-488b-8b31-9ba405917c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr" Nov 03 02:21:40.772268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:40.772238 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr" Nov 03 02:21:41.202651 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:21:41.202616 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33f6aaf_7588_488b_8b31_9ba405917c08.slice/crio-b6fb6e247bc066316bdc2bbd363b6b990f4b65993d44e6041908a78965994d21 WatchSource:0}: Error finding container b6fb6e247bc066316bdc2bbd363b6b990f4b65993d44e6041908a78965994d21: Status 404 returned error can't find the container with id b6fb6e247bc066316bdc2bbd363b6b990f4b65993d44e6041908a78965994d21 Nov 03 02:21:41.205598 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:41.205572 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr"] Nov 03 02:21:41.235173 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:41.235145 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6"] Nov 03 02:21:41.239312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:41.239294 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6" Nov 03 02:21:41.308126 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:41.308099 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6"] Nov 03 02:21:41.397798 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:41.397755 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnbg6\" (UniqueName: \"kubernetes.io/projected/7fdb6b65-5f4e-4058-b392-0539939099fb-kube-api-access-pnbg6\") pod \"payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6\" (UID: \"7fdb6b65-5f4e-4058-b392-0539939099fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6" Nov 03 02:21:41.398007 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:41.397812 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fdb6b65-5f4e-4058-b392-0539939099fb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6\" (UID: \"7fdb6b65-5f4e-4058-b392-0539939099fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6" Nov 03 02:21:41.499092 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:41.498993 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnbg6\" (UniqueName: \"kubernetes.io/projected/7fdb6b65-5f4e-4058-b392-0539939099fb-kube-api-access-pnbg6\") pod \"payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6\" (UID: \"7fdb6b65-5f4e-4058-b392-0539939099fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6" Nov 03 02:21:41.499092 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:41.499041 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fdb6b65-5f4e-4058-b392-0539939099fb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6\" (UID: \"7fdb6b65-5f4e-4058-b392-0539939099fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6" Nov 03 02:21:41.499874 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:41.499849 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fdb6b65-5f4e-4058-b392-0539939099fb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6\" (UID: \"7fdb6b65-5f4e-4058-b392-0539939099fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6" Nov 03 02:21:41.572247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:41.572157 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnbg6\" (UniqueName: \"kubernetes.io/projected/7fdb6b65-5f4e-4058-b392-0539939099fb-kube-api-access-pnbg6\") pod \"payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6\" (UID: \"7fdb6b65-5f4e-4058-b392-0539939099fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6" Nov 03 02:21:41.573501 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:41.573477 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6" Nov 03 02:21:41.601463 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:41.601431 5463 patch_prober.go:29] interesting pod/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 02:21:41.601586 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:41.601493 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" podUID="871baea4-93af-43d2-8ef7-0b05c4f50526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 02:21:41.629249 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:41.629139 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr" event={"ID":"a33f6aaf-7588-488b-8b31-9ba405917c08","Type":"ContainerStarted","Data":"d078d7a04c28d7eeef67a19d3b3dd876d51b1082b5572d0e26b6f72f15dcde6b"} Nov 03 02:21:41.629249 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:41.629207 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr" event={"ID":"a33f6aaf-7588-488b-8b31-9ba405917c08","Type":"ContainerStarted","Data":"b6fb6e247bc066316bdc2bbd363b6b990f4b65993d44e6041908a78965994d21"} Nov 03 02:21:41.629518 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:41.629462 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr" Nov 03 02:21:41.783824 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:41.783355 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr" podStartSLOduration=1.783332348 podStartE2EDuration="1.783332348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:21:41.744277875 +0000 UTC m=+1783.428387031" watchObservedRunningTime="2026-11-03 02:21:41.783332348 +0000 UTC m=+1783.467441504" Nov 03 02:21:41.872656 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:41.872621 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:41.991615 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:21:41.991142 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fdb6b65_5f4e_4058_b392_0539939099fb.slice/crio-a8c110ade07c0d2a2aef487849de0778f66f70c7a2474dc2ea7cb6defb6b0223 WatchSource:0}: Error finding container a8c110ade07c0d2a2aef487849de0778f66f70c7a2474dc2ea7cb6defb6b0223: Status 404 returned error can't find the container with id a8c110ade07c0d2a2aef487849de0778f66f70c7a2474dc2ea7cb6defb6b0223 Nov 03 02:21:41.992987 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:41.992963 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6"] Nov 03 02:21:42.019915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:42.019885 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:42.276568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:42.276533 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:42.601072 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:42.601040 5463 patch_prober.go:29] interesting pod/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 02:21:42.601289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:42.601111 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" podUID="871baea4-93af-43d2-8ef7-0b05c4f50526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 02:21:42.642454 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:42.642410 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6" event={"ID":"7fdb6b65-5f4e-4058-b392-0539939099fb","Type":"ContainerStarted","Data":"dbbd9d9a31746b75f5aaa74830ee3a36e4cb6fb27ae8b85ab66c304453cd4f50"} Nov 03 02:21:42.642670 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:42.642479 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6" event={"ID":"7fdb6b65-5f4e-4058-b392-0539939099fb","Type":"ContainerStarted","Data":"a8c110ade07c0d2a2aef487849de0778f66f70c7a2474dc2ea7cb6defb6b0223"} Nov 03 02:21:42.642670 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:42.642512 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6" Nov 03 02:21:42.761462 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:42.761410 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6" podStartSLOduration=1.7613929069999998 podStartE2EDuration="1.761392907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:21:42.759597587 +0000 UTC m=+1784.443706742" watchObservedRunningTime="2026-11-03 02:21:42.761392907 +0000 UTC m=+1784.445502063" Nov 03 02:21:43.008250 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:21:43.008178 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:21:43.605880 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:43.605847 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:21:43.606233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:43.606212 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:21:43.606917 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:43.606897 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:21:46.739398 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:46.739351 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:48.791153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:48.791118 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:48.944545 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:48.944516 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:51.505656 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:51.505618 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:52.645361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:52.645334 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr" Nov 03 02:21:53.652403 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:53.652371 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6" Nov 03 02:21:56.170824 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:56.170745 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:56.332419 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:56.332381 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:56.479759 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:56.479656 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:57.555618 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:21:57.555583 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:00.017370 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:00.017334 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:00.362369 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:00.362254 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv"] Nov 03 02:22:00.486409 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:00.486371 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg"] Nov 03 02:22:00.726536 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:00.726507 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:00.891873 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:00.890253 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:01.317102 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:01.317070 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:02.634726 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:02.634699 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:22:02.639941 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:02.639920 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:22:05.363820 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.363736 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv" podUID="5549425a-e04c-4436-bf47-def96605061e" containerName="payload-processing" containerID="cri-o://6c3f2e0a6f7ea557b6520ce98410c37f2affaa97e6d1ebe9f8693aa37cec6cd0" gracePeriod=25 Nov 03 02:22:05.487176 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.487133 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg" podUID="4c007625-c8c9-4e25-980c-23d59b9725d6" containerName="payload-pre-processing" containerID="cri-o://0f29cab8f980fa34bd5a3bdb79204b07c444848fca9daf3ee37d82a527a6ba01" gracePeriod=25 Nov 03 02:22:05.658862 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.658831 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:05.665042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.665015 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv" Nov 03 02:22:05.681290 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.681089 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5549425a-e04c-4436-bf47-def96605061e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5549425a-e04c-4436-bf47-def96605061e-plugins-config-volume\") pod \"5549425a-e04c-4436-bf47-def96605061e\" (UID: \"5549425a-e04c-4436-bf47-def96605061e\") " Nov 03 02:22:05.681431 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.681382 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5549425a-e04c-4436-bf47-def96605061e-kube-api-access-nq4n2\" (UniqueName: \"kubernetes.io/projected/5549425a-e04c-4436-bf47-def96605061e-kube-api-access-nq4n2\") pod \"5549425a-e04c-4436-bf47-def96605061e\" (UID: \"5549425a-e04c-4436-bf47-def96605061e\") " Nov 03 02:22:05.682573 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.682531 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5549425a-e04c-4436-bf47-def96605061e-plugins-config-volume" pod "5549425a-e04c-4436-bf47-def96605061e" (UID: "5549425a-e04c-4436-bf47-def96605061e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:22:05.688617 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.688579 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5549425a-e04c-4436-bf47-def96605061e-kube-api-access-nq4n2" pod "5549425a-e04c-4436-bf47-def96605061e" (UID: "5549425a-e04c-4436-bf47-def96605061e"). InnerVolumeSpecName "kube-api-access-nq4n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:22:05.775819 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.775765 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg" Nov 03 02:22:05.782916 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.782889 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq4n2\" (UniqueName: \"kubernetes.io/projected/5549425a-e04c-4436-bf47-def96605061e-kube-api-access-nq4n2\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:05.782916 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.782918 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5549425a-e04c-4436-bf47-def96605061e-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:05.878844 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.878813 5463 generic.go:356] "Generic (PLEG): container finished" podID="943f92d8-5f35-4532-a30e-41961407b68b" containerID="7b628e4470c5c1a7882801a7696f5384c05a11ac808d4fe7a016b03b729a03a7" exitCode=0 Nov 03 02:22:05.879019 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.878898 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" event={"ID":"943f92d8-5f35-4532-a30e-41961407b68b","Type":"ContainerDied","Data":"7b628e4470c5c1a7882801a7696f5384c05a11ac808d4fe7a016b03b729a03a7"} Nov 03 02:22:05.879405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.879377 5463 scope.go:122] "RemoveContainer" containerID="7b628e4470c5c1a7882801a7696f5384c05a11ac808d4fe7a016b03b729a03a7" Nov 03 02:22:05.880621 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.880600 5463 generic.go:356] "Generic (PLEG): container finished" podID="4c007625-c8c9-4e25-980c-23d59b9725d6" containerID="0f29cab8f980fa34bd5a3bdb79204b07c444848fca9daf3ee37d82a527a6ba01" exitCode=0 Nov 03 02:22:05.880754 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.880629 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg" event={"ID":"4c007625-c8c9-4e25-980c-23d59b9725d6","Type":"ContainerDied","Data":"0f29cab8f980fa34bd5a3bdb79204b07c444848fca9daf3ee37d82a527a6ba01"} Nov 03 02:22:05.880754 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.880669 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg" Nov 03 02:22:05.880754 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.880689 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg" event={"ID":"4c007625-c8c9-4e25-980c-23d59b9725d6","Type":"ContainerDied","Data":"72026cdb1872853f519c3cf68707c6d79a276ff139a862f1731fe0ec1e72d3b9"} Nov 03 02:22:05.880754 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.880721 5463 scope.go:122] "RemoveContainer" containerID="0f29cab8f980fa34bd5a3bdb79204b07c444848fca9daf3ee37d82a527a6ba01" Nov 03 02:22:05.881157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.881127 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:22:05.881278 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.881249 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:22:05.881334 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.881300 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:22:05.882620 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.882597 5463 generic.go:356] "Generic (PLEG): container finished" podID="5549425a-e04c-4436-bf47-def96605061e" containerID="6c3f2e0a6f7ea557b6520ce98410c37f2affaa97e6d1ebe9f8693aa37cec6cd0" exitCode=0 Nov 03 02:22:05.882719 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.882648 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv" event={"ID":"5549425a-e04c-4436-bf47-def96605061e","Type":"ContainerDied","Data":"6c3f2e0a6f7ea557b6520ce98410c37f2affaa97e6d1ebe9f8693aa37cec6cd0"} Nov 03 02:22:05.882719 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.882683 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv" event={"ID":"5549425a-e04c-4436-bf47-def96605061e","Type":"ContainerDied","Data":"3a9890df5922d9246be400883008b0dfd9b472048ac245fcc4e6b65e1357db54"} Nov 03 02:22:05.882800 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.882741 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv" Nov 03 02:22:05.883416 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.883386 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c007625-c8c9-4e25-980c-23d59b9725d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c007625-c8c9-4e25-980c-23d59b9725d6-plugins-config-volume\") pod \"4c007625-c8c9-4e25-980c-23d59b9725d6\" (UID: \"4c007625-c8c9-4e25-980c-23d59b9725d6\") " Nov 03 02:22:05.883607 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.883591 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c007625-c8c9-4e25-980c-23d59b9725d6-kube-api-access-kfhsn\" (UniqueName: \"kubernetes.io/projected/4c007625-c8c9-4e25-980c-23d59b9725d6-kube-api-access-kfhsn\") pod \"4c007625-c8c9-4e25-980c-23d59b9725d6\" (UID: \"4c007625-c8c9-4e25-980c-23d59b9725d6\") " Nov 03 02:22:05.884581 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.884553 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c007625-c8c9-4e25-980c-23d59b9725d6-plugins-config-volume" pod "4c007625-c8c9-4e25-980c-23d59b9725d6" (UID: "4c007625-c8c9-4e25-980c-23d59b9725d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:22:05.886127 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.885961 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c007625-c8c9-4e25-980c-23d59b9725d6-kube-api-access-kfhsn" pod "4c007625-c8c9-4e25-980c-23d59b9725d6" (UID: "4c007625-c8c9-4e25-980c-23d59b9725d6"). InnerVolumeSpecName "kube-api-access-kfhsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:22:05.901407 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.901385 5463 scope.go:122] "RemoveContainer" containerID="0f29cab8f980fa34bd5a3bdb79204b07c444848fca9daf3ee37d82a527a6ba01" Nov 03 02:22:05.901805 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:22:05.901783 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f29cab8f980fa34bd5a3bdb79204b07c444848fca9daf3ee37d82a527a6ba01\": container with ID starting with 0f29cab8f980fa34bd5a3bdb79204b07c444848fca9daf3ee37d82a527a6ba01 not found: ID does not exist" containerID="0f29cab8f980fa34bd5a3bdb79204b07c444848fca9daf3ee37d82a527a6ba01" Nov 03 02:22:05.901975 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.901951 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f29cab8f980fa34bd5a3bdb79204b07c444848fca9daf3ee37d82a527a6ba01"} err="failed to get container status \"0f29cab8f980fa34bd5a3bdb79204b07c444848fca9daf3ee37d82a527a6ba01\": rpc error: code = NotFound desc = could not find container \"0f29cab8f980fa34bd5a3bdb79204b07c444848fca9daf3ee37d82a527a6ba01\": container with ID starting with 0f29cab8f980fa34bd5a3bdb79204b07c444848fca9daf3ee37d82a527a6ba01 not found: ID does not exist" Nov 03 02:22:05.902114 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.902102 5463 scope.go:122] "RemoveContainer" containerID="6c3f2e0a6f7ea557b6520ce98410c37f2affaa97e6d1ebe9f8693aa37cec6cd0" Nov 03 02:22:05.924390 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.924363 5463 scope.go:122] "RemoveContainer" containerID="6c3f2e0a6f7ea557b6520ce98410c37f2affaa97e6d1ebe9f8693aa37cec6cd0" Nov 03 02:22:05.924738 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:22:05.924708 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3f2e0a6f7ea557b6520ce98410c37f2affaa97e6d1ebe9f8693aa37cec6cd0\": container with ID starting with 6c3f2e0a6f7ea557b6520ce98410c37f2affaa97e6d1ebe9f8693aa37cec6cd0 not found: ID does not exist" containerID="6c3f2e0a6f7ea557b6520ce98410c37f2affaa97e6d1ebe9f8693aa37cec6cd0" Nov 03 02:22:05.924849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.924743 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3f2e0a6f7ea557b6520ce98410c37f2affaa97e6d1ebe9f8693aa37cec6cd0"} err="failed to get container status \"6c3f2e0a6f7ea557b6520ce98410c37f2affaa97e6d1ebe9f8693aa37cec6cd0\": rpc error: code = NotFound desc = could not find container \"6c3f2e0a6f7ea557b6520ce98410c37f2affaa97e6d1ebe9f8693aa37cec6cd0\": container with ID starting with 6c3f2e0a6f7ea557b6520ce98410c37f2affaa97e6d1ebe9f8693aa37cec6cd0 not found: ID does not exist" Nov 03 02:22:05.984931 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.984907 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfhsn\" (UniqueName: \"kubernetes.io/projected/4c007625-c8c9-4e25-980c-23d59b9725d6-kube-api-access-kfhsn\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:05.985081 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:05.984935 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c007625-c8c9-4e25-980c-23d59b9725d6-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:06.238250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:06.238214 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv"] Nov 03 02:22:06.241386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:06.241357 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5ef2c8-7cdf756d65-mhhdv"] Nov 03 02:22:06.245615 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:06.245572 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:06.350952 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:06.350921 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg"] Nov 03 02:22:06.353995 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:06.353957 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ef2c8-57fbb66bcf-mmldg"] Nov 03 02:22:06.445918 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:06.445841 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c007625-c8c9-4e25-980c-23d59b9725d6" path="/var/lib/kubelet/pods/4c007625-c8c9-4e25-980c-23d59b9725d6/volumes" Nov 03 02:22:06.446345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:06.446329 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5549425a-e04c-4436-bf47-def96605061e" path="/var/lib/kubelet/pods/5549425a-e04c-4436-bf47-def96605061e/volumes" Nov 03 02:22:06.892779 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:06.892689 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" event={"ID":"943f92d8-5f35-4532-a30e-41961407b68b","Type":"ContainerStarted","Data":"e12b42cbc4c20c3a5fd45d787cdb52a706388901c94d3b90e92a1556ca40df0a"} Nov 03 02:22:06.922392 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:06.922369 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:22:06.939579 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:06.939555 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:22:06.941000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:06.940978 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 02:22:06.941119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:06.941020 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" podUID="943f92d8-5f35-4532-a30e-41961407b68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 02:22:07.944218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:07.944138 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 02:22:07.944693 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:07.944399 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" podUID="943f92d8-5f35-4532-a30e-41961407b68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 02:22:08.939855 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:08.939823 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 02:22:08.940032 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:08.939871 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" podUID="943f92d8-5f35-4532-a30e-41961407b68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 02:22:09.939887 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:09.939855 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 02:22:09.940300 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:09.939905 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" podUID="943f92d8-5f35-4532-a30e-41961407b68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 02:22:10.939524 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:10.939486 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 02:22:10.939703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:10.939545 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" podUID="943f92d8-5f35-4532-a30e-41961407b68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 02:22:11.274487 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:11.271775 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:11.939376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:11.939342 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 02:22:11.939570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:11.939393 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" podUID="943f92d8-5f35-4532-a30e-41961407b68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 02:22:12.945001 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:12.944967 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:22:12.946316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:12.946292 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:22:13.011074 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:22:13.011041 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:22:13.307357 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:13.307280 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:15.944474 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:15.944440 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:17.047423 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:17.047388 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw"] Nov 03 02:22:17.097936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:17.097901 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92a46951-cbszw"] Nov 03 02:22:18.446269 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:18.446224 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2d1e8a-89d7-4729-8aed-0176001f4233" path="/var/lib/kubelet/pods/7a2d1e8a-89d7-4729-8aed-0176001f4233/volumes" Nov 03 02:22:19.777810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:19.777748 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:20.652221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:20.652168 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:20.963331 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:20.963250 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:21.154669 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.154628 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22"] Nov 03 02:22:21.155524 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.155499 5463 cpu_manager.go:414] "containerMap: removing container" podUID="5549425a-e04c-4436-bf47-def96605061e" containerName="payload-processing" Nov 03 02:22:21.155672 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.155549 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5549425a-e04c-4436-bf47-def96605061e" containerName="payload-processing" Nov 03 02:22:21.155672 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.155607 5463 cpu_manager.go:414] "containerMap: removing container" podUID="4c007625-c8c9-4e25-980c-23d59b9725d6" containerName="payload-pre-processing" Nov 03 02:22:21.155672 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.155617 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c007625-c8c9-4e25-980c-23d59b9725d6" containerName="payload-pre-processing" Nov 03 02:22:21.155844 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.155803 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="5549425a-e04c-4436-bf47-def96605061e" containerName="payload-processing" Nov 03 02:22:21.155844 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.155831 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c007625-c8c9-4e25-980c-23d59b9725d6" containerName="payload-pre-processing" Nov 03 02:22:21.158656 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.158634 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.212064 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.212034 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d63a25-openshift-default-dockercfg-7vlvm\"" Nov 03 02:22:21.260883 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.260798 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:21.267939 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.267905 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22"] Nov 03 02:22:21.333959 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.333677 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-token\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.334298 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.334271 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-envoy\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.334498 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.334476 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-credential-socket\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.334665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.334647 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-data\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.334803 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.334787 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-workload-certs\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.340871 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.334923 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istiod-ca-cert\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.340871 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.340430 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-podinfo\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.340871 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.340471 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjcnd\" (UniqueName: \"kubernetes.io/projected/b2a15ca1-f5c6-41af-849f-9e727badc8a7-kube-api-access-fjcnd\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.340871 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.340556 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-workload-socket\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.430889 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.430848 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:21.441519 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.441493 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-podinfo\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.441668 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.441529 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjcnd\" (UniqueName: \"kubernetes.io/projected/b2a15ca1-f5c6-41af-849f-9e727badc8a7-kube-api-access-fjcnd\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.441668 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.441593 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-workload-socket\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.441752 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.441675 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-token\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.441752 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.441726 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-envoy\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.441854 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.441754 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-credential-socket\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.441854 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.441807 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-data\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.441854 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.441842 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-workload-certs\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.442027 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.441872 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istiod-ca-cert\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.442092 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.442074 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-workload-socket\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.442326 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.442299 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-credential-socket\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.442436 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.442320 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-data\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.442545 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.442527 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-workload-certs\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.442682 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.442663 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istiod-ca-cert\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.444011 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.443983 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-envoy\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.444117 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.444023 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-podinfo\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.498841 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.498806 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjcnd\" (UniqueName: \"kubernetes.io/projected/b2a15ca1-f5c6-41af-849f-9e727badc8a7-kube-api-access-fjcnd\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.498973 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.498852 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-token\") pod \"e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:21.773538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:21.773501 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:22.195640 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:22:22.195595 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a15ca1_f5c6_41af_849f_9e727badc8a7.slice/crio-9465165317a02d1661bd188ec60013b84d18dcdc858b6379d91c96dec86014cc WatchSource:0}: Error finding container 9465165317a02d1661bd188ec60013b84d18dcdc858b6379d91c96dec86014cc: Status 404 returned error can't find the container with id 9465165317a02d1661bd188ec60013b84d18dcdc858b6379d91c96dec86014cc Nov 03 02:22:22.198364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:22.198323 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:22:22.198490 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:22.198408 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:22:22.198490 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:22.198457 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:22:22.199671 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:22.199650 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22"] Nov 03 02:22:23.026831 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:23.026789 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" event={"ID":"b2a15ca1-f5c6-41af-849f-9e727badc8a7","Type":"ContainerStarted","Data":"53792e0b9d91a6c631fdd5b4462e467a020b9048d53f72f129dd7dd57e8ab792"} Nov 03 02:22:23.027025 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:23.026856 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" event={"ID":"b2a15ca1-f5c6-41af-849f-9e727badc8a7","Type":"ContainerStarted","Data":"9465165317a02d1661bd188ec60013b84d18dcdc858b6379d91c96dec86014cc"} Nov 03 02:22:23.197466 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:23.197310 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" podStartSLOduration=2.197289785 podStartE2EDuration="2.197289785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:22:23.194312167 +0000 UTC m=+1824.878421322" watchObservedRunningTime="2026-11-03 02:22:23.197289785 +0000 UTC m=+1824.881398943" Nov 03 02:22:23.773699 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:23.773670 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:23.779549 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:23.779524 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:24.035102 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:24.035023 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:24.035882 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:24.035859 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:22:26.074159 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:26.074123 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:26.225046 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:26.225000 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:26.373070 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:26.372974 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:26.574174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:26.574136 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:26.781581 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:26.781552 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:26.959624 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:26.959591 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:29.835963 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:29.835882 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:30.004559 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:30.004522 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:31.273500 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:31.273467 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:32.077382 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:32.077338 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:22:32.077630 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:32.077600 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" podUID="255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:22:33.169797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:33.169759 5463 generic.go:356] "Generic (PLEG): container finished" podID="255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" containerID="fe60cace8e3b1fbf70992ffacac504bcb16db23fdbe1286a97c2c387bb6b77ae" exitCode=0 Nov 03 02:22:33.170267 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:33.169832 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" event={"ID":"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e","Type":"ContainerDied","Data":"fe60cace8e3b1fbf70992ffacac504bcb16db23fdbe1286a97c2c387bb6b77ae"} Nov 03 02:22:33.170314 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:33.170295 5463 scope.go:122] "RemoveContainer" containerID="fe60cace8e3b1fbf70992ffacac504bcb16db23fdbe1286a97c2c387bb6b77ae" Nov 03 02:22:33.172020 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:33.171986 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:22:33.172141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:33.172073 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:22:33.172225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:33.172146 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:22:34.182624 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:34.182585 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" event={"ID":"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e","Type":"ContainerStarted","Data":"fb5c65ed4de6baf0dd3d2cdf6d7f2c33abc9343b79e138f1cfb283fa31329399"} Nov 03 02:22:34.714623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:34.714592 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:22:34.714812 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:34.714773 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 03 02:22:34.714904 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:34.714830 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" podUID="255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 03 02:22:35.714770 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:35.714731 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 03 02:22:35.715248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:35.714786 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" podUID="255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 03 02:22:36.147933 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:36.147898 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:36.714345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:36.714305 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 03 02:22:36.714565 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:36.714366 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" podUID="255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 03 02:22:37.715002 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:37.714966 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 03 02:22:37.715616 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:37.715030 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" podUID="255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 03 02:22:38.714260 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:38.714226 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 03 02:22:38.714460 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:38.714296 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" podUID="255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 03 02:22:39.718848 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:39.718818 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:22:39.719333 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:39.719298 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:22:39.719947 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:39.719928 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:22:41.448311 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:41.448277 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:41.771943 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:41.771858 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls"] Nov 03 02:22:41.772124 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:41.772099 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" podUID="0d549eff-db94-49f8-bff2-4e7f7d24071d" containerName="maas-api" containerID="cri-o://43129c8669d6dbe19e05bf587dbc75470d6e9c93874509ea5ce8209f24bf131a" gracePeriod=30 Nov 03 02:22:41.893561 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:41.893529 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:42.073080 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.073052 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" Nov 03 02:22:42.169000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.168962 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d549eff-db94-49f8-bff2-4e7f7d24071d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d549eff-db94-49f8-bff2-4e7f7d24071d-maas-api-tls\") pod \"0d549eff-db94-49f8-bff2-4e7f7d24071d\" (UID: \"0d549eff-db94-49f8-bff2-4e7f7d24071d\") " Nov 03 02:22:42.169178 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.169153 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d549eff-db94-49f8-bff2-4e7f7d24071d-kube-api-access-plthn\" (UniqueName: \"kubernetes.io/projected/0d549eff-db94-49f8-bff2-4e7f7d24071d-kube-api-access-plthn\") pod \"0d549eff-db94-49f8-bff2-4e7f7d24071d\" (UID: \"0d549eff-db94-49f8-bff2-4e7f7d24071d\") " Nov 03 02:22:42.169264 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.169222 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d549eff-db94-49f8-bff2-4e7f7d24071d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d549eff-db94-49f8-bff2-4e7f7d24071d-metrics-certs\") pod \"0d549eff-db94-49f8-bff2-4e7f7d24071d\" (UID: \"0d549eff-db94-49f8-bff2-4e7f7d24071d\") " Nov 03 02:22:42.169322 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.169300 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d549eff-db94-49f8-bff2-4e7f7d24071d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d549eff-db94-49f8-bff2-4e7f7d24071d-openshift-service-ca\") pod \"0d549eff-db94-49f8-bff2-4e7f7d24071d\" (UID: \"0d549eff-db94-49f8-bff2-4e7f7d24071d\") " Nov 03 02:22:42.169896 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.169845 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d549eff-db94-49f8-bff2-4e7f7d24071d-openshift-service-ca" pod "0d549eff-db94-49f8-bff2-4e7f7d24071d" (UID: "0d549eff-db94-49f8-bff2-4e7f7d24071d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:22:42.171523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.171503 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d549eff-db94-49f8-bff2-4e7f7d24071d-metrics-certs" pod "0d549eff-db94-49f8-bff2-4e7f7d24071d" (UID: "0d549eff-db94-49f8-bff2-4e7f7d24071d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:22:42.171799 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.171770 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d549eff-db94-49f8-bff2-4e7f7d24071d-maas-api-tls" pod "0d549eff-db94-49f8-bff2-4e7f7d24071d" (UID: "0d549eff-db94-49f8-bff2-4e7f7d24071d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:22:42.171799 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.171780 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d549eff-db94-49f8-bff2-4e7f7d24071d-kube-api-access-plthn" pod "0d549eff-db94-49f8-bff2-4e7f7d24071d" (UID: "0d549eff-db94-49f8-bff2-4e7f7d24071d"). InnerVolumeSpecName "kube-api-access-plthn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:22:42.215789 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.215755 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc"] Nov 03 02:22:42.216034 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.216001 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc" podUID="39a6a726-254a-4e88-9f5e-de47741cc446" containerName="payload-processing" containerID="cri-o://2cf604e9081f9c20b8d5dc2cc7b4593f95569dea64de99f64ef067209a991712" gracePeriod=30 Nov 03 02:22:42.255362 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.255317 5463 generic.go:356] "Generic (PLEG): container finished" podID="0d549eff-db94-49f8-bff2-4e7f7d24071d" containerID="43129c8669d6dbe19e05bf587dbc75470d6e9c93874509ea5ce8209f24bf131a" exitCode=0 Nov 03 02:22:42.255543 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.255485 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" event={"ID":"0d549eff-db94-49f8-bff2-4e7f7d24071d","Type":"ContainerDied","Data":"43129c8669d6dbe19e05bf587dbc75470d6e9c93874509ea5ce8209f24bf131a"} Nov 03 02:22:42.255543 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.255538 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" event={"ID":"0d549eff-db94-49f8-bff2-4e7f7d24071d","Type":"ContainerDied","Data":"e60dec01c86a2faa9134492e91d248417c1d26ba2e17295b8d307479a07b48c2"} Nov 03 02:22:42.255668 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.255568 5463 scope.go:122] "RemoveContainer" containerID="43129c8669d6dbe19e05bf587dbc75470d6e9c93874509ea5ce8209f24bf131a" Nov 03 02:22:42.255747 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.255721 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls" Nov 03 02:22:42.270892 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.270867 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f"] Nov 03 02:22:42.271065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.270879 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plthn\" (UniqueName: \"kubernetes.io/projected/0d549eff-db94-49f8-bff2-4e7f7d24071d-kube-api-access-plthn\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:42.271161 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.271080 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d549eff-db94-49f8-bff2-4e7f7d24071d-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:42.271161 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.271097 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d549eff-db94-49f8-bff2-4e7f7d24071d-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:42.271161 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.271111 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d549eff-db94-49f8-bff2-4e7f7d24071d-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:42.271531 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.271259 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f" podUID="e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98" containerName="payload-pre-processing" containerID="cri-o://3bae839164142af27a779d0c6c56f5c2e016160916b7bec8bd954609afdd9aec" gracePeriod=30 Nov 03 02:22:42.345159 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.345135 5463 scope.go:122] "RemoveContainer" containerID="43129c8669d6dbe19e05bf587dbc75470d6e9c93874509ea5ce8209f24bf131a" Nov 03 02:22:42.345533 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:22:42.345511 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43129c8669d6dbe19e05bf587dbc75470d6e9c93874509ea5ce8209f24bf131a\": container with ID starting with 43129c8669d6dbe19e05bf587dbc75470d6e9c93874509ea5ce8209f24bf131a not found: ID does not exist" containerID="43129c8669d6dbe19e05bf587dbc75470d6e9c93874509ea5ce8209f24bf131a" Nov 03 02:22:42.345627 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.345543 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43129c8669d6dbe19e05bf587dbc75470d6e9c93874509ea5ce8209f24bf131a"} err="failed to get container status \"43129c8669d6dbe19e05bf587dbc75470d6e9c93874509ea5ce8209f24bf131a\": rpc error: code = NotFound desc = could not find container \"43129c8669d6dbe19e05bf587dbc75470d6e9c93874509ea5ce8209f24bf131a\": container with ID starting with 43129c8669d6dbe19e05bf587dbc75470d6e9c93874509ea5ce8209f24bf131a not found: ID does not exist" Nov 03 02:22:42.401250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.399514 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:42.543411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.543376 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:42.544168 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.544148 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc" Nov 03 02:22:42.569786 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.569751 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls"] Nov 03 02:22:42.573585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.573563 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ff26f5-545976bdbd-d85ls"] Nov 03 02:22:42.578411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.578389 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39a6a726-254a-4e88-9f5e-de47741cc446-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a6a726-254a-4e88-9f5e-de47741cc446-plugins-config-volume\") pod \"39a6a726-254a-4e88-9f5e-de47741cc446\" (UID: \"39a6a726-254a-4e88-9f5e-de47741cc446\") " Nov 03 02:22:42.578542 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.578493 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a6a726-254a-4e88-9f5e-de47741cc446-kube-api-access-hq644\" (UniqueName: \"kubernetes.io/projected/39a6a726-254a-4e88-9f5e-de47741cc446-kube-api-access-hq644\") pod \"39a6a726-254a-4e88-9f5e-de47741cc446\" (UID: \"39a6a726-254a-4e88-9f5e-de47741cc446\") " Nov 03 02:22:42.578785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.578762 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39a6a726-254a-4e88-9f5e-de47741cc446-plugins-config-volume" pod "39a6a726-254a-4e88-9f5e-de47741cc446" (UID: "39a6a726-254a-4e88-9f5e-de47741cc446"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:22:42.578973 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.578955 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a6a726-254a-4e88-9f5e-de47741cc446-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:42.580623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.580597 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a6a726-254a-4e88-9f5e-de47741cc446-kube-api-access-hq644" pod "39a6a726-254a-4e88-9f5e-de47741cc446" (UID: "39a6a726-254a-4e88-9f5e-de47741cc446"). InnerVolumeSpecName "kube-api-access-hq644". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:22:42.597808 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.597755 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f" Nov 03 02:22:42.680059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.680029 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98-kube-api-access-pj4lw\" (UniqueName: \"kubernetes.io/projected/e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98-kube-api-access-pj4lw\") pod \"e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98\" (UID: \"e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98\") " Nov 03 02:22:42.680447 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.680107 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98-plugins-config-volume\") pod \"e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98\" (UID: \"e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98\") " Nov 03 02:22:42.680780 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.680748 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq644\" (UniqueName: \"kubernetes.io/projected/39a6a726-254a-4e88-9f5e-de47741cc446-kube-api-access-hq644\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:42.680896 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.680865 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98-plugins-config-volume" pod "e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98" (UID: "e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:22:42.683114 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.683090 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98-kube-api-access-pj4lw" pod "e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98" (UID: "e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98"). InnerVolumeSpecName "kube-api-access-pj4lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:22:42.784322 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.784283 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj4lw\" (UniqueName: \"kubernetes.io/projected/e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98-kube-api-access-pj4lw\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:42.784565 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:42.784546 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:43.008317 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:22:43.008280 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:22:43.264346 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:43.264252 5463 generic.go:356] "Generic (PLEG): container finished" podID="e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98" containerID="3bae839164142af27a779d0c6c56f5c2e016160916b7bec8bd954609afdd9aec" exitCode=0 Nov 03 02:22:43.264346 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:43.264304 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f" Nov 03 02:22:43.264346 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:43.264335 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f" event={"ID":"e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98","Type":"ContainerDied","Data":"3bae839164142af27a779d0c6c56f5c2e016160916b7bec8bd954609afdd9aec"} Nov 03 02:22:43.264642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:43.264422 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f" event={"ID":"e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98","Type":"ContainerDied","Data":"a79fbde7716b5054495a3bb46cb1061eefbbafceee740c9e62c9e3b081f94f32"} Nov 03 02:22:43.264642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:43.264453 5463 scope.go:122] "RemoveContainer" containerID="3bae839164142af27a779d0c6c56f5c2e016160916b7bec8bd954609afdd9aec" Nov 03 02:22:43.266130 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:43.266105 5463 generic.go:356] "Generic (PLEG): container finished" podID="39a6a726-254a-4e88-9f5e-de47741cc446" containerID="2cf604e9081f9c20b8d5dc2cc7b4593f95569dea64de99f64ef067209a991712" exitCode=0 Nov 03 02:22:43.266307 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:43.266147 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc" Nov 03 02:22:43.266307 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:43.266185 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc" event={"ID":"39a6a726-254a-4e88-9f5e-de47741cc446","Type":"ContainerDied","Data":"2cf604e9081f9c20b8d5dc2cc7b4593f95569dea64de99f64ef067209a991712"} Nov 03 02:22:43.266307 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:43.266273 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc" event={"ID":"39a6a726-254a-4e88-9f5e-de47741cc446","Type":"ContainerDied","Data":"bf8866d2b8904cd2931439ec1c36869bebb4f4142f053923bfccb435d19463d2"} Nov 03 02:22:43.280354 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:43.280333 5463 scope.go:122] "RemoveContainer" containerID="3bae839164142af27a779d0c6c56f5c2e016160916b7bec8bd954609afdd9aec" Nov 03 02:22:43.280653 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:22:43.280629 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bae839164142af27a779d0c6c56f5c2e016160916b7bec8bd954609afdd9aec\": container with ID starting with 3bae839164142af27a779d0c6c56f5c2e016160916b7bec8bd954609afdd9aec not found: ID does not exist" containerID="3bae839164142af27a779d0c6c56f5c2e016160916b7bec8bd954609afdd9aec" Nov 03 02:22:43.280742 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:43.280661 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bae839164142af27a779d0c6c56f5c2e016160916b7bec8bd954609afdd9aec"} err="failed to get container status \"3bae839164142af27a779d0c6c56f5c2e016160916b7bec8bd954609afdd9aec\": rpc error: code = NotFound desc = could not find container \"3bae839164142af27a779d0c6c56f5c2e016160916b7bec8bd954609afdd9aec\": container with ID starting with 3bae839164142af27a779d0c6c56f5c2e016160916b7bec8bd954609afdd9aec not found: ID does not exist" Nov 03 02:22:43.280742 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:43.280677 5463 scope.go:122] "RemoveContainer" containerID="2cf604e9081f9c20b8d5dc2cc7b4593f95569dea64de99f64ef067209a991712" Nov 03 02:22:43.290836 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:43.290820 5463 scope.go:122] "RemoveContainer" containerID="2cf604e9081f9c20b8d5dc2cc7b4593f95569dea64de99f64ef067209a991712" Nov 03 02:22:43.291080 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:22:43.291063 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf604e9081f9c20b8d5dc2cc7b4593f95569dea64de99f64ef067209a991712\": container with ID starting with 2cf604e9081f9c20b8d5dc2cc7b4593f95569dea64de99f64ef067209a991712 not found: ID does not exist" containerID="2cf604e9081f9c20b8d5dc2cc7b4593f95569dea64de99f64ef067209a991712" Nov 03 02:22:43.291135 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:43.291087 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf604e9081f9c20b8d5dc2cc7b4593f95569dea64de99f64ef067209a991712"} err="failed to get container status \"2cf604e9081f9c20b8d5dc2cc7b4593f95569dea64de99f64ef067209a991712\": rpc error: code = NotFound desc = could not find container \"2cf604e9081f9c20b8d5dc2cc7b4593f95569dea64de99f64ef067209a991712\": container with ID starting with 2cf604e9081f9c20b8d5dc2cc7b4593f95569dea64de99f64ef067209a991712 not found: ID does not exist" Nov 03 02:22:43.429354 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:43.429325 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f"] Nov 03 02:22:43.432368 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:43.432339 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ff26f5-7cd5dd8cbc-5l28f"] Nov 03 02:22:43.540377 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:43.540299 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc"] Nov 03 02:22:43.543410 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:43.543381 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ff26f5-7dc6d64bcb-2twnc"] Nov 03 02:22:43.858417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:43.858289 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:44.444262 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:44.444231 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d549eff-db94-49f8-bff2-4e7f7d24071d" path="/var/lib/kubelet/pods/0d549eff-db94-49f8-bff2-4e7f7d24071d/volumes" Nov 03 02:22:44.444821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:44.444802 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39a6a726-254a-4e88-9f5e-de47741cc446" path="/var/lib/kubelet/pods/39a6a726-254a-4e88-9f5e-de47741cc446/volumes" Nov 03 02:22:44.445242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:44.445229 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98" path="/var/lib/kubelet/pods/e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98/volumes" Nov 03 02:22:45.871146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:45.871111 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:46.031057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:46.031022 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:47.465214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:47.465142 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:47.974703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:47.974666 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:47.975253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:47.975226 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk"] Nov 03 02:22:47.975505 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:47.975475 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" podUID="a6d496ad-cf03-4990-9326-a127a83750e2" containerName="istio-proxy" containerID="cri-o://a520af000a0d527663206474894eaa3441e97fe2ec4909b281566a865096fceb" gracePeriod=30 Nov 03 02:22:48.780239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:48.779698 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:50.630660 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:50.630619 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:53.271739 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.271675 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:22:53.348868 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.348832 5463 generic.go:356] "Generic (PLEG): container finished" podID="a6d496ad-cf03-4990-9326-a127a83750e2" containerID="a520af000a0d527663206474894eaa3441e97fe2ec4909b281566a865096fceb" exitCode=0 Nov 03 02:22:53.349021 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.348907 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" Nov 03 02:22:53.349021 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.348913 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" event={"ID":"a6d496ad-cf03-4990-9326-a127a83750e2","Type":"ContainerDied","Data":"a520af000a0d527663206474894eaa3441e97fe2ec4909b281566a865096fceb"} Nov 03 02:22:53.349141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.349031 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk" event={"ID":"a6d496ad-cf03-4990-9326-a127a83750e2","Type":"ContainerDied","Data":"d6c25f3b9ec4b970a6e00d268c9c0481634d6e3be4ec08c4054676978cc10bd3"} Nov 03 02:22:53.349141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.349056 5463 scope.go:122] "RemoveContainer" containerID="a520af000a0d527663206474894eaa3441e97fe2ec4909b281566a865096fceb" Nov 03 02:22:53.362758 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.362738 5463 scope.go:122] "RemoveContainer" containerID="2d5583ec3f7631bc8c0ff3725312d77683e20eacfffc051fcba9943bb2b725c2" Nov 03 02:22:53.372516 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.372496 5463 scope.go:122] "RemoveContainer" containerID="a520af000a0d527663206474894eaa3441e97fe2ec4909b281566a865096fceb" Nov 03 02:22:53.372840 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:22:53.372807 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a520af000a0d527663206474894eaa3441e97fe2ec4909b281566a865096fceb\": container with ID starting with a520af000a0d527663206474894eaa3441e97fe2ec4909b281566a865096fceb not found: ID does not exist" containerID="a520af000a0d527663206474894eaa3441e97fe2ec4909b281566a865096fceb" Nov 03 02:22:53.372954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.372850 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a520af000a0d527663206474894eaa3441e97fe2ec4909b281566a865096fceb"} err="failed to get container status \"a520af000a0d527663206474894eaa3441e97fe2ec4909b281566a865096fceb\": rpc error: code = NotFound desc = could not find container \"a520af000a0d527663206474894eaa3441e97fe2ec4909b281566a865096fceb\": container with ID starting with a520af000a0d527663206474894eaa3441e97fe2ec4909b281566a865096fceb not found: ID does not exist" Nov 03 02:22:53.372954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.372873 5463 scope.go:122] "RemoveContainer" containerID="2d5583ec3f7631bc8c0ff3725312d77683e20eacfffc051fcba9943bb2b725c2" Nov 03 02:22:53.373216 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:22:53.373163 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d5583ec3f7631bc8c0ff3725312d77683e20eacfffc051fcba9943bb2b725c2\": container with ID starting with 2d5583ec3f7631bc8c0ff3725312d77683e20eacfffc051fcba9943bb2b725c2 not found: ID does not exist" containerID="2d5583ec3f7631bc8c0ff3725312d77683e20eacfffc051fcba9943bb2b725c2" Nov 03 02:22:53.373291 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.373232 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d5583ec3f7631bc8c0ff3725312d77683e20eacfffc051fcba9943bb2b725c2"} err="failed to get container status \"2d5583ec3f7631bc8c0ff3725312d77683e20eacfffc051fcba9943bb2b725c2\": rpc error: code = NotFound desc = could not find container \"2d5583ec3f7631bc8c0ff3725312d77683e20eacfffc051fcba9943bb2b725c2\": container with ID starting with 2d5583ec3f7631bc8c0ff3725312d77683e20eacfffc051fcba9943bb2b725c2 not found: ID does not exist" Nov 03 02:22:53.378559 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.378534 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-istio-data\") pod \"a6d496ad-cf03-4990-9326-a127a83750e2\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " Nov 03 02:22:53.378637 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.378591 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6d496ad-cf03-4990-9326-a127a83750e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6d496ad-cf03-4990-9326-a127a83750e2-istio-podinfo\") pod \"a6d496ad-cf03-4990-9326-a127a83750e2\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " Nov 03 02:22:53.378691 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.378672 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6d496ad-cf03-4990-9326-a127a83750e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6d496ad-cf03-4990-9326-a127a83750e2-istiod-ca-cert\") pod \"a6d496ad-cf03-4990-9326-a127a83750e2\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " Nov 03 02:22:53.378762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.378745 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-workload-socket\") pod \"a6d496ad-cf03-4990-9326-a127a83750e2\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " Nov 03 02:22:53.378856 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.378831 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-istio-data" pod "a6d496ad-cf03-4990-9326-a127a83750e2" (UID: "a6d496ad-cf03-4990-9326-a127a83750e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:22:53.378916 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.378865 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d496ad-cf03-4990-9326-a127a83750e2-istio-token\" (UniqueName: \"kubernetes.io/projected/a6d496ad-cf03-4990-9326-a127a83750e2-istio-token\") pod \"a6d496ad-cf03-4990-9326-a127a83750e2\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " Nov 03 02:22:53.378974 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.378917 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d496ad-cf03-4990-9326-a127a83750e2-kube-api-access-6c725\" (UniqueName: \"kubernetes.io/projected/a6d496ad-cf03-4990-9326-a127a83750e2-kube-api-access-6c725\") pod \"a6d496ad-cf03-4990-9326-a127a83750e2\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " Nov 03 02:22:53.378974 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.378946 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-credential-socket\") pod \"a6d496ad-cf03-4990-9326-a127a83750e2\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " Nov 03 02:22:53.379087 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.379033 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-workload-certs\") pod \"a6d496ad-cf03-4990-9326-a127a83750e2\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " Nov 03 02:22:53.379087 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.379052 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-workload-socket" pod "a6d496ad-cf03-4990-9326-a127a83750e2" (UID: "a6d496ad-cf03-4990-9326-a127a83750e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:22:53.379087 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.379068 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-istio-envoy\") pod \"a6d496ad-cf03-4990-9326-a127a83750e2\" (UID: \"a6d496ad-cf03-4990-9326-a127a83750e2\") " Nov 03 02:22:53.379308 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.379283 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-credential-socket" pod "a6d496ad-cf03-4990-9326-a127a83750e2" (UID: "a6d496ad-cf03-4990-9326-a127a83750e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:22:53.379308 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.379297 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-workload-certs" pod "a6d496ad-cf03-4990-9326-a127a83750e2" (UID: "a6d496ad-cf03-4990-9326-a127a83750e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:22:53.379458 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.379365 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d496ad-cf03-4990-9326-a127a83750e2-istiod-ca-cert" pod "a6d496ad-cf03-4990-9326-a127a83750e2" (UID: "a6d496ad-cf03-4990-9326-a127a83750e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:22:53.379635 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.379605 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:53.379635 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.379630 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:53.379815 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.379645 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:53.379815 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.379658 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6d496ad-cf03-4990-9326-a127a83750e2-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:53.379815 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.379672 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:53.381633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.381611 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d496ad-cf03-4990-9326-a127a83750e2-kube-api-access-6c725" pod "a6d496ad-cf03-4990-9326-a127a83750e2" (UID: "a6d496ad-cf03-4990-9326-a127a83750e2"). InnerVolumeSpecName "kube-api-access-6c725". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:22:53.381788 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.381765 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-istio-envoy" pod "a6d496ad-cf03-4990-9326-a127a83750e2" (UID: "a6d496ad-cf03-4990-9326-a127a83750e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:22:53.381872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.381824 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6d496ad-cf03-4990-9326-a127a83750e2-istio-podinfo" pod "a6d496ad-cf03-4990-9326-a127a83750e2" (UID: "a6d496ad-cf03-4990-9326-a127a83750e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:22:53.382248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.382221 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d496ad-cf03-4990-9326-a127a83750e2-istio-token" pod "a6d496ad-cf03-4990-9326-a127a83750e2" (UID: "a6d496ad-cf03-4990-9326-a127a83750e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:22:53.481816 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.481783 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6d496ad-cf03-4990-9326-a127a83750e2-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:53.481816 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.481815 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6d496ad-cf03-4990-9326-a127a83750e2-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:53.482109 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.481834 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c725\" (UniqueName: \"kubernetes.io/projected/a6d496ad-cf03-4990-9326-a127a83750e2-kube-api-access-6c725\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:53.482109 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.481847 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6d496ad-cf03-4990-9326-a127a83750e2-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:53.773660 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.773625 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk"] Nov 03 02:22:53.778472 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:53.778445 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ff26f5-openshift-default-5fcb44b5bf-ppplk"] Nov 03 02:22:54.360565 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:54.360475 5463 generic.go:356] "Generic (PLEG): container finished" podID="a7d39617-aba6-4951-b3ce-14f544abbf7a" containerID="42e9941f147f6c7e61d605cdffa98cc6a0b7707baa76c6853888e3011bd2fa7e" exitCode=0 Nov 03 02:22:54.360565 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:54.360549 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" event={"ID":"a7d39617-aba6-4951-b3ce-14f544abbf7a","Type":"ContainerDied","Data":"42e9941f147f6c7e61d605cdffa98cc6a0b7707baa76c6853888e3011bd2fa7e"} Nov 03 02:22:54.361072 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:54.361029 5463 scope.go:122] "RemoveContainer" containerID="42e9941f147f6c7e61d605cdffa98cc6a0b7707baa76c6853888e3011bd2fa7e" Nov 03 02:22:54.362734 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:54.362696 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:22:54.362870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:54.362771 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:22:54.362870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:54.362805 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:22:54.443459 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:54.443419 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d496ad-cf03-4990-9326-a127a83750e2" path="/var/lib/kubelet/pods/a6d496ad-cf03-4990-9326-a127a83750e2/volumes" Nov 03 02:22:54.676540 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:54.676504 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:54.896215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:54.893293 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:55.383218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:55.380610 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" event={"ID":"a7d39617-aba6-4951-b3ce-14f544abbf7a","Type":"ContainerStarted","Data":"9aaf1ece518e1c2b57ec4658d0c7bd01954314ba84c4c9050d4c3b1e72d2cf03"} Nov 03 02:22:55.623294 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:55.623256 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:56.313706 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:56.313676 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:22:56.314762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:56.314730 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 03 02:22:56.314903 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:56.314779 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" podUID="a7d39617-aba6-4951-b3ce-14f544abbf7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 03 02:22:57.313775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:57.313739 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 03 02:22:57.314284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:57.313798 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" podUID="a7d39617-aba6-4951-b3ce-14f544abbf7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 03 02:22:58.313905 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:58.313872 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 03 02:22:58.314353 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:58.313944 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" podUID="a7d39617-aba6-4951-b3ce-14f544abbf7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 03 02:22:59.313996 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:59.313954 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 03 02:22:59.314514 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:59.314022 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" podUID="a7d39617-aba6-4951-b3ce-14f544abbf7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 03 02:22:59.339243 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:22:59.339215 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:23:00.026386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.026353 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw"] Nov 03 02:23:00.026597 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.026573 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" podUID="feb8fd27-a4ea-41d8-ba16-69279ff9e85b" containerName="maas-api" containerID="cri-o://509014614ad31824df579c81e33a136482ca54b36b2c52f493b471242d292d6d" gracePeriod=30 Nov 03 02:23:00.174218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.173064 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:00.315076 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.314984 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 03 02:23:00.315619 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.315063 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" podUID="a7d39617-aba6-4951-b3ce-14f544abbf7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 03 02:23:00.375229 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.375183 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" Nov 03 02:23:00.422573 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.422540 5463 generic.go:356] "Generic (PLEG): container finished" podID="feb8fd27-a4ea-41d8-ba16-69279ff9e85b" containerID="509014614ad31824df579c81e33a136482ca54b36b2c52f493b471242d292d6d" exitCode=0 Nov 03 02:23:00.422759 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.422601 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" Nov 03 02:23:00.422759 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.422631 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" event={"ID":"feb8fd27-a4ea-41d8-ba16-69279ff9e85b","Type":"ContainerDied","Data":"509014614ad31824df579c81e33a136482ca54b36b2c52f493b471242d292d6d"} Nov 03 02:23:00.422759 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.422688 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw" event={"ID":"feb8fd27-a4ea-41d8-ba16-69279ff9e85b","Type":"ContainerDied","Data":"822529bc576f96043af79ca09c87850947748f19badae127dba91f5f19eec707"} Nov 03 02:23:00.422759 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.422710 5463 scope.go:122] "RemoveContainer" containerID="509014614ad31824df579c81e33a136482ca54b36b2c52f493b471242d292d6d" Nov 03 02:23:00.491617 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.472573 5463 scope.go:122] "RemoveContainer" containerID="509014614ad31824df579c81e33a136482ca54b36b2c52f493b471242d292d6d" Nov 03 02:23:00.492170 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:23:00.492129 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"509014614ad31824df579c81e33a136482ca54b36b2c52f493b471242d292d6d\": container with ID starting with 509014614ad31824df579c81e33a136482ca54b36b2c52f493b471242d292d6d not found: ID does not exist" containerID="509014614ad31824df579c81e33a136482ca54b36b2c52f493b471242d292d6d" Nov 03 02:23:00.492320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.492182 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509014614ad31824df579c81e33a136482ca54b36b2c52f493b471242d292d6d"} err="failed to get container status \"509014614ad31824df579c81e33a136482ca54b36b2c52f493b471242d292d6d\": rpc error: code = NotFound desc = could not find container \"509014614ad31824df579c81e33a136482ca54b36b2c52f493b471242d292d6d\": container with ID starting with 509014614ad31824df579c81e33a136482ca54b36b2c52f493b471242d292d6d not found: ID does not exist" Nov 03 02:23:00.506949 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.506671 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-openshift-service-ca\") pod \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\" (UID: \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\") " Nov 03 02:23:00.508082 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.508053 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-kube-api-access-nw8jc\" (UniqueName: \"kubernetes.io/projected/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-kube-api-access-nw8jc\") pod \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\" (UID: \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\") " Nov 03 02:23:00.508755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.507312 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-openshift-service-ca" pod "feb8fd27-a4ea-41d8-ba16-69279ff9e85b" (UID: "feb8fd27-a4ea-41d8-ba16-69279ff9e85b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:23:00.509204 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.509168 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-metrics-certs\") pod \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\" (UID: \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\") " Nov 03 02:23:00.509513 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.509493 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-maas-api-tls\") pod \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\" (UID: \"feb8fd27-a4ea-41d8-ba16-69279ff9e85b\") " Nov 03 02:23:00.510605 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.510583 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:00.514598 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.514562 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-metrics-certs" pod "feb8fd27-a4ea-41d8-ba16-69279ff9e85b" (UID: "feb8fd27-a4ea-41d8-ba16-69279ff9e85b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:23:00.514701 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.514617 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-kube-api-access-nw8jc" pod "feb8fd27-a4ea-41d8-ba16-69279ff9e85b" (UID: "feb8fd27-a4ea-41d8-ba16-69279ff9e85b"). InnerVolumeSpecName "kube-api-access-nw8jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:23:00.515242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.515210 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-maas-api-tls" pod "feb8fd27-a4ea-41d8-ba16-69279ff9e85b" (UID: "feb8fd27-a4ea-41d8-ba16-69279ff9e85b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:23:00.612574 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.612491 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:00.612574 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.612520 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw8jc\" (UniqueName: \"kubernetes.io/projected/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-kube-api-access-nw8jc\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:00.612574 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.612535 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feb8fd27-a4ea-41d8-ba16-69279ff9e85b-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:00.848660 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.848628 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw"] Nov 03 02:23:00.852590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:00.852560 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ef2c8-6d7486b567-zwdpw"] Nov 03 02:23:01.318964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:01.318929 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:23:01.320121 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:01.320101 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:23:01.425728 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:01.425695 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk"] Nov 03 02:23:01.489819 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:01.489778 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf"] Nov 03 02:23:01.639354 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:01.639272 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:01.803215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:01.803145 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:01.948919 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:01.948883 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:02.101301 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:02.101270 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:02.444240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:02.444206 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feb8fd27-a4ea-41d8-ba16-69279ff9e85b" path="/var/lib/kubelet/pods/feb8fd27-a4ea-41d8-ba16-69279ff9e85b/volumes" Nov 03 02:23:06.426809 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:06.426767 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk" podUID="2152640c-b513-4397-b448-dbd5199d6aaa" containerName="payload-processing" containerID="cri-o://caf08aa6b8709b7bb5945061f630aca74b5ec629c4ca0f664655cb6b182a8c38" gracePeriod=25 Nov 03 02:23:06.490387 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:06.490352 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf" podUID="aaff3f9e-3094-4e97-863e-c832964b6a61" containerName="payload-pre-processing" containerID="cri-o://130b1331525bb518e1ed30ee98f9a62bea276527537cfcd117e7030ff767628e" gracePeriod=25 Nov 03 02:23:06.856447 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:06.856087 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf" Nov 03 02:23:06.899755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:06.899731 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk" Nov 03 02:23:06.994966 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:06.994897 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2152640c-b513-4397-b448-dbd5199d6aaa-kube-api-access-5dvlq\" (UniqueName: \"kubernetes.io/projected/2152640c-b513-4397-b448-dbd5199d6aaa-kube-api-access-5dvlq\") pod \"2152640c-b513-4397-b448-dbd5199d6aaa\" (UID: \"2152640c-b513-4397-b448-dbd5199d6aaa\") " Nov 03 02:23:06.995095 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:06.995022 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaff3f9e-3094-4e97-863e-c832964b6a61-kube-api-access-pwnzd\" (UniqueName: \"kubernetes.io/projected/aaff3f9e-3094-4e97-863e-c832964b6a61-kube-api-access-pwnzd\") pod \"aaff3f9e-3094-4e97-863e-c832964b6a61\" (UID: \"aaff3f9e-3094-4e97-863e-c832964b6a61\") " Nov 03 02:23:06.995137 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:06.995118 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaff3f9e-3094-4e97-863e-c832964b6a61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaff3f9e-3094-4e97-863e-c832964b6a61-plugins-config-volume\") pod \"aaff3f9e-3094-4e97-863e-c832964b6a61\" (UID: \"aaff3f9e-3094-4e97-863e-c832964b6a61\") " Nov 03 02:23:06.995180 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:06.995161 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2152640c-b513-4397-b448-dbd5199d6aaa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2152640c-b513-4397-b448-dbd5199d6aaa-plugins-config-volume\") pod \"2152640c-b513-4397-b448-dbd5199d6aaa\" (UID: \"2152640c-b513-4397-b448-dbd5199d6aaa\") " Nov 03 02:23:06.996088 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:06.995974 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaff3f9e-3094-4e97-863e-c832964b6a61-plugins-config-volume" pod "aaff3f9e-3094-4e97-863e-c832964b6a61" (UID: "aaff3f9e-3094-4e97-863e-c832964b6a61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:23:06.996088 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:06.996056 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2152640c-b513-4397-b448-dbd5199d6aaa-plugins-config-volume" pod "2152640c-b513-4397-b448-dbd5199d6aaa" (UID: "2152640c-b513-4397-b448-dbd5199d6aaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:23:06.997905 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:06.997884 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2152640c-b513-4397-b448-dbd5199d6aaa-kube-api-access-5dvlq" pod "2152640c-b513-4397-b448-dbd5199d6aaa" (UID: "2152640c-b513-4397-b448-dbd5199d6aaa"). InnerVolumeSpecName "kube-api-access-5dvlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:23:06.998014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:06.997957 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaff3f9e-3094-4e97-863e-c832964b6a61-kube-api-access-pwnzd" pod "aaff3f9e-3094-4e97-863e-c832964b6a61" (UID: "aaff3f9e-3094-4e97-863e-c832964b6a61"). InnerVolumeSpecName "kube-api-access-pwnzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:23:07.097007 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.096964 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dvlq\" (UniqueName: \"kubernetes.io/projected/2152640c-b513-4397-b448-dbd5199d6aaa-kube-api-access-5dvlq\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:07.097007 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.097007 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwnzd\" (UniqueName: \"kubernetes.io/projected/aaff3f9e-3094-4e97-863e-c832964b6a61-kube-api-access-pwnzd\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:07.097291 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.097024 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaff3f9e-3094-4e97-863e-c832964b6a61-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:07.097291 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.097039 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2152640c-b513-4397-b448-dbd5199d6aaa-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:07.382586 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.382501 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:07.480364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.480334 5463 generic.go:356] "Generic (PLEG): container finished" podID="aaff3f9e-3094-4e97-863e-c832964b6a61" containerID="130b1331525bb518e1ed30ee98f9a62bea276527537cfcd117e7030ff767628e" exitCode=0 Nov 03 02:23:07.480788 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.480368 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf" Nov 03 02:23:07.480788 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.480424 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf" event={"ID":"aaff3f9e-3094-4e97-863e-c832964b6a61","Type":"ContainerDied","Data":"130b1331525bb518e1ed30ee98f9a62bea276527537cfcd117e7030ff767628e"} Nov 03 02:23:07.480788 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.480483 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf" event={"ID":"aaff3f9e-3094-4e97-863e-c832964b6a61","Type":"ContainerDied","Data":"80c2f89cc3283f433582184ff68c761029c7d37358fced16966388a07272473b"} Nov 03 02:23:07.480788 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.480512 5463 scope.go:122] "RemoveContainer" containerID="130b1331525bb518e1ed30ee98f9a62bea276527537cfcd117e7030ff767628e" Nov 03 02:23:07.482302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.482270 5463 generic.go:356] "Generic (PLEG): container finished" podID="2152640c-b513-4397-b448-dbd5199d6aaa" containerID="caf08aa6b8709b7bb5945061f630aca74b5ec629c4ca0f664655cb6b182a8c38" exitCode=0 Nov 03 02:23:07.482443 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.482306 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk" Nov 03 02:23:07.482443 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.482351 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk" event={"ID":"2152640c-b513-4397-b448-dbd5199d6aaa","Type":"ContainerDied","Data":"caf08aa6b8709b7bb5945061f630aca74b5ec629c4ca0f664655cb6b182a8c38"} Nov 03 02:23:07.482443 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.482399 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk" event={"ID":"2152640c-b513-4397-b448-dbd5199d6aaa","Type":"ContainerDied","Data":"8d83353e3a60c001ee2d3f087e471c9e82be8ad64c3c7fb3c3b47ec64bf631b7"} Nov 03 02:23:07.503834 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.503814 5463 scope.go:122] "RemoveContainer" containerID="130b1331525bb518e1ed30ee98f9a62bea276527537cfcd117e7030ff767628e" Nov 03 02:23:07.504143 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:23:07.504120 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"130b1331525bb518e1ed30ee98f9a62bea276527537cfcd117e7030ff767628e\": container with ID starting with 130b1331525bb518e1ed30ee98f9a62bea276527537cfcd117e7030ff767628e not found: ID does not exist" containerID="130b1331525bb518e1ed30ee98f9a62bea276527537cfcd117e7030ff767628e" Nov 03 02:23:07.504224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.504150 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"130b1331525bb518e1ed30ee98f9a62bea276527537cfcd117e7030ff767628e"} err="failed to get container status \"130b1331525bb518e1ed30ee98f9a62bea276527537cfcd117e7030ff767628e\": rpc error: code = NotFound desc = could not find container \"130b1331525bb518e1ed30ee98f9a62bea276527537cfcd117e7030ff767628e\": container with ID starting with 130b1331525bb518e1ed30ee98f9a62bea276527537cfcd117e7030ff767628e not found: ID does not exist" Nov 03 02:23:07.504224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.504168 5463 scope.go:122] "RemoveContainer" containerID="caf08aa6b8709b7bb5945061f630aca74b5ec629c4ca0f664655cb6b182a8c38" Nov 03 02:23:07.514645 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.514628 5463 scope.go:122] "RemoveContainer" containerID="caf08aa6b8709b7bb5945061f630aca74b5ec629c4ca0f664655cb6b182a8c38" Nov 03 02:23:07.514886 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:23:07.514868 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caf08aa6b8709b7bb5945061f630aca74b5ec629c4ca0f664655cb6b182a8c38\": container with ID starting with caf08aa6b8709b7bb5945061f630aca74b5ec629c4ca0f664655cb6b182a8c38 not found: ID does not exist" containerID="caf08aa6b8709b7bb5945061f630aca74b5ec629c4ca0f664655cb6b182a8c38" Nov 03 02:23:07.514951 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.514892 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf08aa6b8709b7bb5945061f630aca74b5ec629c4ca0f664655cb6b182a8c38"} err="failed to get container status \"caf08aa6b8709b7bb5945061f630aca74b5ec629c4ca0f664655cb6b182a8c38\": rpc error: code = NotFound desc = could not find container \"caf08aa6b8709b7bb5945061f630aca74b5ec629c4ca0f664655cb6b182a8c38\": container with ID starting with caf08aa6b8709b7bb5945061f630aca74b5ec629c4ca0f664655cb6b182a8c38 not found: ID does not exist" Nov 03 02:23:07.646290 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.646243 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk"] Nov 03 02:23:07.651632 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.651606 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-84b9bcb57-8fqbk"] Nov 03 02:23:07.760662 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.760626 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf"] Nov 03 02:23:07.765205 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:07.765171 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-65cb6776dc-92rpf"] Nov 03 02:23:08.444009 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:08.443977 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2152640c-b513-4397-b448-dbd5199d6aaa" path="/var/lib/kubelet/pods/2152640c-b513-4397-b448-dbd5199d6aaa/volumes" Nov 03 02:23:08.444420 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:08.444405 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaff3f9e-3094-4e97-863e-c832964b6a61" path="/var/lib/kubelet/pods/aaff3f9e-3094-4e97-863e-c832964b6a61/volumes" Nov 03 02:23:08.761827 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:08.761718 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:08.915600 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:08.915569 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:10.275893 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:10.275858 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:10.880127 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:10.880087 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d"] Nov 03 02:23:10.880365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:10.880338 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" podUID="ba02d63a-95b8-4195-b07d-4f0e68398761" containerName="istio-proxy" containerID="cri-o://faace85c23203886a42f5d8b71ea3f476a418b0f9b99794aefde5e320446646a" gracePeriod=30 Nov 03 02:23:10.883216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:10.883169 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:11.619566 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:11.619537 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686"] Nov 03 02:23:11.723801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:11.723768 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5"] Nov 03 02:23:11.885081 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:11.884976 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:12.036555 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.036509 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:12.182771 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.182741 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:12.453280 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.453168 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:12.605268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.605235 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:12.980716 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.980684 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj"] Nov 03 02:23:12.981806 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.981782 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a6d496ad-cf03-4990-9326-a127a83750e2" containerName="istio-proxy" Nov 03 02:23:12.981806 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.981804 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d496ad-cf03-4990-9326-a127a83750e2" containerName="istio-proxy" Nov 03 02:23:12.982012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.981816 5463 cpu_manager.go:414] "containerMap: removing container" podUID="39a6a726-254a-4e88-9f5e-de47741cc446" containerName="payload-processing" Nov 03 02:23:12.982012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.981821 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39a6a726-254a-4e88-9f5e-de47741cc446" containerName="payload-processing" Nov 03 02:23:12.982012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.981852 5463 cpu_manager.go:414] "containerMap: removing container" podUID="feb8fd27-a4ea-41d8-ba16-69279ff9e85b" containerName="maas-api" Nov 03 02:23:12.982012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.981860 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feb8fd27-a4ea-41d8-ba16-69279ff9e85b" containerName="maas-api" Nov 03 02:23:12.982012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.981878 5463 cpu_manager.go:414] "containerMap: removing container" podUID="0d549eff-db94-49f8-bff2-4e7f7d24071d" containerName="maas-api" Nov 03 02:23:12.982012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.981883 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d549eff-db94-49f8-bff2-4e7f7d24071d" containerName="maas-api" Nov 03 02:23:12.982012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.981896 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a6d496ad-cf03-4990-9326-a127a83750e2" containerName="istio-proxy" Nov 03 02:23:12.982012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.981901 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d496ad-cf03-4990-9326-a127a83750e2" containerName="istio-proxy" Nov 03 02:23:12.982012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.981910 5463 cpu_manager.go:414] "containerMap: removing container" podUID="e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98" containerName="payload-pre-processing" Nov 03 02:23:12.982012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.981915 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98" containerName="payload-pre-processing" Nov 03 02:23:12.982012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.981932 5463 cpu_manager.go:414] "containerMap: removing container" podUID="aaff3f9e-3094-4e97-863e-c832964b6a61" containerName="payload-pre-processing" Nov 03 02:23:12.982012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.981939 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaff3f9e-3094-4e97-863e-c832964b6a61" containerName="payload-pre-processing" Nov 03 02:23:12.982012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.981951 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2152640c-b513-4397-b448-dbd5199d6aaa" containerName="payload-processing" Nov 03 02:23:12.982012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.981955 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2152640c-b513-4397-b448-dbd5199d6aaa" containerName="payload-processing" Nov 03 02:23:12.982797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.982061 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93d28f4-8a13-4d8a-a6a5-8d5bdb516f98" containerName="payload-pre-processing" Nov 03 02:23:12.982797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.982073 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaff3f9e-3094-4e97-863e-c832964b6a61" containerName="payload-pre-processing" Nov 03 02:23:12.982797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.982085 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2152640c-b513-4397-b448-dbd5199d6aaa" containerName="payload-processing" Nov 03 02:23:12.982797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.982096 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d496ad-cf03-4990-9326-a127a83750e2" containerName="istio-proxy" Nov 03 02:23:12.982797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.982102 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="39a6a726-254a-4e88-9f5e-de47741cc446" containerName="payload-processing" Nov 03 02:23:12.982797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.982112 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="feb8fd27-a4ea-41d8-ba16-69279ff9e85b" containerName="maas-api" Nov 03 02:23:12.982797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.982119 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d549eff-db94-49f8-bff2-4e7f7d24071d" containerName="maas-api" Nov 03 02:23:12.982797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.982126 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d496ad-cf03-4990-9326-a127a83750e2" containerName="istio-proxy" Nov 03 02:23:12.984996 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:12.984967 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.014823 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:23:13.014788 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:23:13.037950 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.037914 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c62c3d-openshift-default-dockercfg-d6f2v\"" Nov 03 02:23:13.069834 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.069805 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-podinfo\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.070002 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.069883 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-token\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.070002 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.069954 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-workload-certs\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.070294 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.070018 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-envoy\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.070294 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.070065 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-workload-socket\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.070294 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.070139 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5ssv\" (UniqueName: \"kubernetes.io/projected/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-kube-api-access-k5ssv\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.071003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.070426 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istiod-ca-cert\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.071003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.070469 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-data\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.071003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.070547 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-credential-socket\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.091390 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.091363 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj"] Nov 03 02:23:13.093332 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.093305 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:13.182514 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.182468 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-token\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.182728 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.182527 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-workload-certs\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.182728 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.182558 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-envoy\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.182728 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.182600 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-workload-socket\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.182728 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.182638 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5ssv\" (UniqueName: \"kubernetes.io/projected/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-kube-api-access-k5ssv\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.182970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.182765 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istiod-ca-cert\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.182970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.182790 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-data\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.182970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.182838 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-credential-socket\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.182970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.182917 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-podinfo\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.189305 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.183726 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-workload-socket\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.189305 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.184340 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-workload-certs\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.189305 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.185337 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istiod-ca-cert\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.189305 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.185764 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-data\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.189305 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.185975 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-credential-socket\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.192083 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.191799 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-podinfo\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.192083 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.192005 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-envoy\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.241391 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.241321 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-token\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.245483 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.245457 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5ssv\" (UniqueName: \"kubernetes.io/projected/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-kube-api-access-k5ssv\") pod \"e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.303835 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.303804 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:13.484015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.483976 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:13.650050 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.649989 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:13.739237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.739206 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj"] Nov 03 02:23:13.771476 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:23:13.771446 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d34a09_e4c1_4298_b62d_7dec1e3f9e1f.slice/crio-4696123f335b0a6b7977a3e573556e0dcb393718b136741a187bd674928837b0 WatchSource:0}: Error finding container 4696123f335b0a6b7977a3e573556e0dcb393718b136741a187bd674928837b0: Status 404 returned error can't find the container with id 4696123f335b0a6b7977a3e573556e0dcb393718b136741a187bd674928837b0 Nov 03 02:23:13.774204 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.774159 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:23:13.774309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.774257 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:23:13.774309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:13.774290 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:23:14.506161 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:14.506122 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:23:14.506558 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:14.506185 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" podUID="ba02d63a-95b8-4195-b07d-4f0e68398761" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:23:14.534275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:14.534241 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" event={"ID":"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f","Type":"ContainerStarted","Data":"ec0e3cbe1669b07275fd6906e388407cd2fac1893c9dba0801903c5c8bf8f3e0"} Nov 03 02:23:14.534444 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:14.534302 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" event={"ID":"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f","Type":"ContainerStarted","Data":"4696123f335b0a6b7977a3e573556e0dcb393718b136741a187bd674928837b0"} Nov 03 02:23:14.680777 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:14.680725 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" podStartSLOduration=2.680712341 podStartE2EDuration="2.680712341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:23:14.678017791 +0000 UTC m=+1876.362126946" watchObservedRunningTime="2026-11-03 02:23:14.680712341 +0000 UTC m=+1876.364821495" Nov 03 02:23:15.304465 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:15.304428 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:15.310440 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:15.310409 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:15.541784 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:15.541754 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:15.542681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:15.542661 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:23:16.159143 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.159116 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:23:16.219840 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.219807 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-credential-socket\") pod \"ba02d63a-95b8-4195-b07d-4f0e68398761\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " Nov 03 02:23:16.220013 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.219872 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-token\" (UniqueName: \"kubernetes.io/projected/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-token\") pod \"ba02d63a-95b8-4195-b07d-4f0e68398761\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " Nov 03 02:23:16.220013 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.219930 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-workload-socket\") pod \"ba02d63a-95b8-4195-b07d-4f0e68398761\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " Nov 03 02:23:16.220013 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.219976 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-workload-certs\") pod \"ba02d63a-95b8-4195-b07d-4f0e68398761\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " Nov 03 02:23:16.220013 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.220011 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba02d63a-95b8-4195-b07d-4f0e68398761-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba02d63a-95b8-4195-b07d-4f0e68398761-istiod-ca-cert\") pod \"ba02d63a-95b8-4195-b07d-4f0e68398761\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " Nov 03 02:23:16.220241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.220046 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-data\") pod \"ba02d63a-95b8-4195-b07d-4f0e68398761\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " Nov 03 02:23:16.220241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.220074 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-envoy\") pod \"ba02d63a-95b8-4195-b07d-4f0e68398761\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " Nov 03 02:23:16.220241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.220117 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-podinfo\") pod \"ba02d63a-95b8-4195-b07d-4f0e68398761\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " Nov 03 02:23:16.220241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.220183 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba02d63a-95b8-4195-b07d-4f0e68398761-kube-api-access-2fwft\" (UniqueName: \"kubernetes.io/projected/ba02d63a-95b8-4195-b07d-4f0e68398761-kube-api-access-2fwft\") pod \"ba02d63a-95b8-4195-b07d-4f0e68398761\" (UID: \"ba02d63a-95b8-4195-b07d-4f0e68398761\") " Nov 03 02:23:16.220465 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.220042 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-credential-socket" pod "ba02d63a-95b8-4195-b07d-4f0e68398761" (UID: "ba02d63a-95b8-4195-b07d-4f0e68398761"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:16.220465 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.220325 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-workload-certs" pod "ba02d63a-95b8-4195-b07d-4f0e68398761" (UID: "ba02d63a-95b8-4195-b07d-4f0e68398761"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:16.220465 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.220352 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-data" pod "ba02d63a-95b8-4195-b07d-4f0e68398761" (UID: "ba02d63a-95b8-4195-b07d-4f0e68398761"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:16.220465 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.220311 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-workload-socket" pod "ba02d63a-95b8-4195-b07d-4f0e68398761" (UID: "ba02d63a-95b8-4195-b07d-4f0e68398761"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:16.220673 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.220607 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:16.220673 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.220633 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:16.220673 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.220651 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:16.220673 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.220664 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:16.220882 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.220676 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba02d63a-95b8-4195-b07d-4f0e68398761-istiod-ca-cert" pod "ba02d63a-95b8-4195-b07d-4f0e68398761" (UID: "ba02d63a-95b8-4195-b07d-4f0e68398761"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:23:16.222910 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.222876 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-envoy" pod "ba02d63a-95b8-4195-b07d-4f0e68398761" (UID: "ba02d63a-95b8-4195-b07d-4f0e68398761"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:16.223029 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.222950 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-podinfo" pod "ba02d63a-95b8-4195-b07d-4f0e68398761" (UID: "ba02d63a-95b8-4195-b07d-4f0e68398761"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:23:16.223029 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.223002 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-token" pod "ba02d63a-95b8-4195-b07d-4f0e68398761" (UID: "ba02d63a-95b8-4195-b07d-4f0e68398761"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:23:16.223111 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.223031 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba02d63a-95b8-4195-b07d-4f0e68398761-kube-api-access-2fwft" pod "ba02d63a-95b8-4195-b07d-4f0e68398761" (UID: "ba02d63a-95b8-4195-b07d-4f0e68398761"). InnerVolumeSpecName "kube-api-access-2fwft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:23:16.321750 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.321722 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:16.321750 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.321747 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba02d63a-95b8-4195-b07d-4f0e68398761-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:16.321951 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.321758 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:16.321951 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.321767 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba02d63a-95b8-4195-b07d-4f0e68398761-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:16.321951 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.321778 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fwft\" (UniqueName: \"kubernetes.io/projected/ba02d63a-95b8-4195-b07d-4f0e68398761-kube-api-access-2fwft\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:16.549858 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.549829 5463 generic.go:356] "Generic (PLEG): container finished" podID="ba02d63a-95b8-4195-b07d-4f0e68398761" containerID="faace85c23203886a42f5d8b71ea3f476a418b0f9b99794aefde5e320446646a" exitCode=0 Nov 03 02:23:16.550348 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.549876 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" Nov 03 02:23:16.550348 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.549919 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" event={"ID":"ba02d63a-95b8-4195-b07d-4f0e68398761","Type":"ContainerDied","Data":"faace85c23203886a42f5d8b71ea3f476a418b0f9b99794aefde5e320446646a"} Nov 03 02:23:16.550348 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.549979 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d" event={"ID":"ba02d63a-95b8-4195-b07d-4f0e68398761","Type":"ContainerDied","Data":"e8df50909b83dc0b4f3d39b57d60192212e923e5ac62f1e69fec59973a3d3701"} Nov 03 02:23:16.550348 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.550019 5463 scope.go:122] "RemoveContainer" containerID="faace85c23203886a42f5d8b71ea3f476a418b0f9b99794aefde5e320446646a" Nov 03 02:23:16.560493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.560464 5463 scope.go:122] "RemoveContainer" containerID="66b1b195d206e7e6365a1b490e570eba02f228ca6191131438bfde3c3e70e576" Nov 03 02:23:16.569618 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.569602 5463 scope.go:122] "RemoveContainer" containerID="faace85c23203886a42f5d8b71ea3f476a418b0f9b99794aefde5e320446646a" Nov 03 02:23:16.569924 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:23:16.569906 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faace85c23203886a42f5d8b71ea3f476a418b0f9b99794aefde5e320446646a\": container with ID starting with faace85c23203886a42f5d8b71ea3f476a418b0f9b99794aefde5e320446646a not found: ID does not exist" containerID="faace85c23203886a42f5d8b71ea3f476a418b0f9b99794aefde5e320446646a" Nov 03 02:23:16.569970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.569935 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faace85c23203886a42f5d8b71ea3f476a418b0f9b99794aefde5e320446646a"} err="failed to get container status \"faace85c23203886a42f5d8b71ea3f476a418b0f9b99794aefde5e320446646a\": rpc error: code = NotFound desc = could not find container \"faace85c23203886a42f5d8b71ea3f476a418b0f9b99794aefde5e320446646a\": container with ID starting with faace85c23203886a42f5d8b71ea3f476a418b0f9b99794aefde5e320446646a not found: ID does not exist" Nov 03 02:23:16.569970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.569952 5463 scope.go:122] "RemoveContainer" containerID="66b1b195d206e7e6365a1b490e570eba02f228ca6191131438bfde3c3e70e576" Nov 03 02:23:16.570248 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:23:16.570231 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b1b195d206e7e6365a1b490e570eba02f228ca6191131438bfde3c3e70e576\": container with ID starting with 66b1b195d206e7e6365a1b490e570eba02f228ca6191131438bfde3c3e70e576 not found: ID does not exist" containerID="66b1b195d206e7e6365a1b490e570eba02f228ca6191131438bfde3c3e70e576" Nov 03 02:23:16.570293 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.570257 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b1b195d206e7e6365a1b490e570eba02f228ca6191131438bfde3c3e70e576"} err="failed to get container status \"66b1b195d206e7e6365a1b490e570eba02f228ca6191131438bfde3c3e70e576\": rpc error: code = NotFound desc = could not find container \"66b1b195d206e7e6365a1b490e570eba02f228ca6191131438bfde3c3e70e576\": container with ID starting with 66b1b195d206e7e6365a1b490e570eba02f228ca6191131438bfde3c3e70e576 not found: ID does not exist" Nov 03 02:23:16.620718 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.620687 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686" podUID="3837cc6f-67ae-413a-ad00-3f2782fc8c5e" containerName="payload-processing" containerID="cri-o://d4ddeee50937b125e831958c26c56b246558697490bcba17287666725a468666" gracePeriod=25 Nov 03 02:23:16.715242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.715163 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d"] Nov 03 02:23:16.718379 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.718358 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5ef2c8-openshift-default-5d9dfbd55f-9kk2d"] Nov 03 02:23:16.724428 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.724400 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5" podUID="86e79ca8-6f1f-421d-b8f8-343394abf780" containerName="payload-pre-processing" containerID="cri-o://b19d15b504fcdb913c5eb9326e5bd544ecd0f18d0ca8dd18f05ffc5ee6162d9d" gracePeriod=25 Nov 03 02:23:16.903520 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.903271 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686" Nov 03 02:23:16.929250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.928714 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3837cc6f-67ae-413a-ad00-3f2782fc8c5e-kube-api-access-gk4m9\" (UniqueName: \"kubernetes.io/projected/3837cc6f-67ae-413a-ad00-3f2782fc8c5e-kube-api-access-gk4m9\") pod \"3837cc6f-67ae-413a-ad00-3f2782fc8c5e\" (UID: \"3837cc6f-67ae-413a-ad00-3f2782fc8c5e\") " Nov 03 02:23:16.929250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.928804 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3837cc6f-67ae-413a-ad00-3f2782fc8c5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3837cc6f-67ae-413a-ad00-3f2782fc8c5e-plugins-config-volume\") pod \"3837cc6f-67ae-413a-ad00-3f2782fc8c5e\" (UID: \"3837cc6f-67ae-413a-ad00-3f2782fc8c5e\") " Nov 03 02:23:16.929590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.929567 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3837cc6f-67ae-413a-ad00-3f2782fc8c5e-plugins-config-volume" pod "3837cc6f-67ae-413a-ad00-3f2782fc8c5e" (UID: "3837cc6f-67ae-413a-ad00-3f2782fc8c5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:23:16.933374 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:16.933348 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3837cc6f-67ae-413a-ad00-3f2782fc8c5e-kube-api-access-gk4m9" pod "3837cc6f-67ae-413a-ad00-3f2782fc8c5e" (UID: "3837cc6f-67ae-413a-ad00-3f2782fc8c5e"). InnerVolumeSpecName "kube-api-access-gk4m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:23:17.029036 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.029010 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5" Nov 03 02:23:17.030383 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.030360 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk4m9\" (UniqueName: \"kubernetes.io/projected/3837cc6f-67ae-413a-ad00-3f2782fc8c5e-kube-api-access-gk4m9\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:17.030449 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.030390 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3837cc6f-67ae-413a-ad00-3f2782fc8c5e-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:17.131635 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.131598 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86e79ca8-6f1f-421d-b8f8-343394abf780-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86e79ca8-6f1f-421d-b8f8-343394abf780-plugins-config-volume\") pod \"86e79ca8-6f1f-421d-b8f8-343394abf780\" (UID: \"86e79ca8-6f1f-421d-b8f8-343394abf780\") " Nov 03 02:23:17.131804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.131647 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e79ca8-6f1f-421d-b8f8-343394abf780-kube-api-access-j9qh5\" (UniqueName: \"kubernetes.io/projected/86e79ca8-6f1f-421d-b8f8-343394abf780-kube-api-access-j9qh5\") pod \"86e79ca8-6f1f-421d-b8f8-343394abf780\" (UID: \"86e79ca8-6f1f-421d-b8f8-343394abf780\") " Nov 03 02:23:17.132013 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.131970 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86e79ca8-6f1f-421d-b8f8-343394abf780-plugins-config-volume" pod "86e79ca8-6f1f-421d-b8f8-343394abf780" (UID: "86e79ca8-6f1f-421d-b8f8-343394abf780"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:23:17.133654 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.133633 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e79ca8-6f1f-421d-b8f8-343394abf780-kube-api-access-j9qh5" pod "86e79ca8-6f1f-421d-b8f8-343394abf780" (UID: "86e79ca8-6f1f-421d-b8f8-343394abf780"). InnerVolumeSpecName "kube-api-access-j9qh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:23:17.232868 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.232835 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86e79ca8-6f1f-421d-b8f8-343394abf780-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:17.232868 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.232864 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9qh5\" (UniqueName: \"kubernetes.io/projected/86e79ca8-6f1f-421d-b8f8-343394abf780-kube-api-access-j9qh5\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:17.509357 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.509310 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:17.559785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.559748 5463 generic.go:356] "Generic (PLEG): container finished" podID="86e79ca8-6f1f-421d-b8f8-343394abf780" containerID="b19d15b504fcdb913c5eb9326e5bd544ecd0f18d0ca8dd18f05ffc5ee6162d9d" exitCode=0 Nov 03 02:23:17.560270 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.559816 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5" Nov 03 02:23:17.560270 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.559831 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5" event={"ID":"86e79ca8-6f1f-421d-b8f8-343394abf780","Type":"ContainerDied","Data":"b19d15b504fcdb913c5eb9326e5bd544ecd0f18d0ca8dd18f05ffc5ee6162d9d"} Nov 03 02:23:17.560270 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.559908 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5" event={"ID":"86e79ca8-6f1f-421d-b8f8-343394abf780","Type":"ContainerDied","Data":"ccab718d8ec2d3eb5fa57d3fc8517aa6df28dae76a0fd448c07f9641b6ad99ca"} Nov 03 02:23:17.560270 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.559956 5463 scope.go:122] "RemoveContainer" containerID="b19d15b504fcdb913c5eb9326e5bd544ecd0f18d0ca8dd18f05ffc5ee6162d9d" Nov 03 02:23:17.565567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.565517 5463 generic.go:356] "Generic (PLEG): container finished" podID="3837cc6f-67ae-413a-ad00-3f2782fc8c5e" containerID="d4ddeee50937b125e831958c26c56b246558697490bcba17287666725a468666" exitCode=0 Nov 03 02:23:17.565567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.565561 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686" Nov 03 02:23:17.565752 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.565588 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686" event={"ID":"3837cc6f-67ae-413a-ad00-3f2782fc8c5e","Type":"ContainerDied","Data":"d4ddeee50937b125e831958c26c56b246558697490bcba17287666725a468666"} Nov 03 02:23:17.565752 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.565626 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686" event={"ID":"3837cc6f-67ae-413a-ad00-3f2782fc8c5e","Type":"ContainerDied","Data":"40017cb8c11f383b35d4b636f5f4209f330fcb10d121062d8a9f7687b1d24093"} Nov 03 02:23:17.573758 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.573730 5463 scope.go:122] "RemoveContainer" containerID="b19d15b504fcdb913c5eb9326e5bd544ecd0f18d0ca8dd18f05ffc5ee6162d9d" Nov 03 02:23:17.574100 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:23:17.574074 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b19d15b504fcdb913c5eb9326e5bd544ecd0f18d0ca8dd18f05ffc5ee6162d9d\": container with ID starting with b19d15b504fcdb913c5eb9326e5bd544ecd0f18d0ca8dd18f05ffc5ee6162d9d not found: ID does not exist" containerID="b19d15b504fcdb913c5eb9326e5bd544ecd0f18d0ca8dd18f05ffc5ee6162d9d" Nov 03 02:23:17.574301 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.574112 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b19d15b504fcdb913c5eb9326e5bd544ecd0f18d0ca8dd18f05ffc5ee6162d9d"} err="failed to get container status \"b19d15b504fcdb913c5eb9326e5bd544ecd0f18d0ca8dd18f05ffc5ee6162d9d\": rpc error: code = NotFound desc = could not find container \"b19d15b504fcdb913c5eb9326e5bd544ecd0f18d0ca8dd18f05ffc5ee6162d9d\": container with ID starting with b19d15b504fcdb913c5eb9326e5bd544ecd0f18d0ca8dd18f05ffc5ee6162d9d not found: ID does not exist" Nov 03 02:23:17.574301 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.574131 5463 scope.go:122] "RemoveContainer" containerID="d4ddeee50937b125e831958c26c56b246558697490bcba17287666725a468666" Nov 03 02:23:17.590172 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.590153 5463 scope.go:122] "RemoveContainer" containerID="d4ddeee50937b125e831958c26c56b246558697490bcba17287666725a468666" Nov 03 02:23:17.590544 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:23:17.590519 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ddeee50937b125e831958c26c56b246558697490bcba17287666725a468666\": container with ID starting with d4ddeee50937b125e831958c26c56b246558697490bcba17287666725a468666 not found: ID does not exist" containerID="d4ddeee50937b125e831958c26c56b246558697490bcba17287666725a468666" Nov 03 02:23:17.590630 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.590555 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ddeee50937b125e831958c26c56b246558697490bcba17287666725a468666"} err="failed to get container status \"d4ddeee50937b125e831958c26c56b246558697490bcba17287666725a468666\": rpc error: code = NotFound desc = could not find container \"d4ddeee50937b125e831958c26c56b246558697490bcba17287666725a468666\": container with ID starting with d4ddeee50937b125e831958c26c56b246558697490bcba17287666725a468666 not found: ID does not exist" Nov 03 02:23:17.730394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.730357 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686"] Nov 03 02:23:17.733509 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.733484 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad4591-855c86d85f-t8686"] Nov 03 02:23:17.735708 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.735687 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:17.857268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.856240 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5"] Nov 03 02:23:17.866063 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.866035 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6fcdf65f9-g5cw5"] Nov 03 02:23:17.883752 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:17.883723 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:18.445156 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:18.445117 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3837cc6f-67ae-413a-ad00-3f2782fc8c5e" path="/var/lib/kubelet/pods/3837cc6f-67ae-413a-ad00-3f2782fc8c5e/volumes" Nov 03 02:23:18.445548 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:18.445531 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86e79ca8-6f1f-421d-b8f8-343394abf780" path="/var/lib/kubelet/pods/86e79ca8-6f1f-421d-b8f8-343394abf780/volumes" Nov 03 02:23:18.445957 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:18.445943 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba02d63a-95b8-4195-b07d-4f0e68398761" path="/var/lib/kubelet/pods/ba02d63a-95b8-4195-b07d-4f0e68398761/volumes" Nov 03 02:23:20.734169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:20.734132 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:20.908122 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:20.908088 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:21.122059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.121968 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp"] Nov 03 02:23:21.122676 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.122659 5463 cpu_manager.go:414] "containerMap: removing container" podUID="3837cc6f-67ae-413a-ad00-3f2782fc8c5e" containerName="payload-processing" Nov 03 02:23:21.122733 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.122677 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3837cc6f-67ae-413a-ad00-3f2782fc8c5e" containerName="payload-processing" Nov 03 02:23:21.122733 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.122711 5463 cpu_manager.go:414] "containerMap: removing container" podUID="ba02d63a-95b8-4195-b07d-4f0e68398761" containerName="istio-proxy" Nov 03 02:23:21.122733 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.122716 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba02d63a-95b8-4195-b07d-4f0e68398761" containerName="istio-proxy" Nov 03 02:23:21.122733 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.122729 5463 cpu_manager.go:414] "containerMap: removing container" podUID="86e79ca8-6f1f-421d-b8f8-343394abf780" containerName="payload-pre-processing" Nov 03 02:23:21.122879 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.122735 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e79ca8-6f1f-421d-b8f8-343394abf780" containerName="payload-pre-processing" Nov 03 02:23:21.122879 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.122772 5463 cpu_manager.go:414] "containerMap: removing container" podUID="ba02d63a-95b8-4195-b07d-4f0e68398761" containerName="istio-proxy" Nov 03 02:23:21.122879 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.122777 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba02d63a-95b8-4195-b07d-4f0e68398761" containerName="istio-proxy" Nov 03 02:23:21.122879 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.122862 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba02d63a-95b8-4195-b07d-4f0e68398761" containerName="istio-proxy" Nov 03 02:23:21.122879 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.122871 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="3837cc6f-67ae-413a-ad00-3f2782fc8c5e" containerName="payload-processing" Nov 03 02:23:21.123020 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.122883 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba02d63a-95b8-4195-b07d-4f0e68398761" containerName="istio-proxy" Nov 03 02:23:21.123020 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.122892 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="86e79ca8-6f1f-421d-b8f8-343394abf780" containerName="payload-pre-processing" Nov 03 02:23:21.125373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.125357 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp" Nov 03 02:23:21.178833 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.178805 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp"] Nov 03 02:23:21.184745 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.184683 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4dg6\" (UniqueName: \"kubernetes.io/projected/0c862e45-235b-4ee4-9efd-094389f6f071-kube-api-access-s4dg6\") pod \"payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp\" (UID: \"0c862e45-235b-4ee4-9efd-094389f6f071\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp" Nov 03 02:23:21.184880 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.184820 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c862e45-235b-4ee4-9efd-094389f6f071-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp\" (UID: \"0c862e45-235b-4ee4-9efd-094389f6f071\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp" Nov 03 02:23:21.240214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.239512 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c62c3d\"" Nov 03 02:23:21.240214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.239697 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c62c3d-dockercfg-b84sm\"" Nov 03 02:23:21.281729 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.280297 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv"] Nov 03 02:23:21.286556 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.286527 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv" Nov 03 02:23:21.290029 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.289426 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4dg6\" (UniqueName: \"kubernetes.io/projected/0c862e45-235b-4ee4-9efd-094389f6f071-kube-api-access-s4dg6\") pod \"payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp\" (UID: \"0c862e45-235b-4ee4-9efd-094389f6f071\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp" Nov 03 02:23:21.290029 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.289495 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c862e45-235b-4ee4-9efd-094389f6f071-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp\" (UID: \"0c862e45-235b-4ee4-9efd-094389f6f071\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp" Nov 03 02:23:21.290471 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.290447 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c862e45-235b-4ee4-9efd-094389f6f071-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp\" (UID: \"0c862e45-235b-4ee4-9efd-094389f6f071\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp" Nov 03 02:23:21.338601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.338570 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv"] Nov 03 02:23:21.348760 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.348727 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4dg6\" (UniqueName: \"kubernetes.io/projected/0c862e45-235b-4ee4-9efd-094389f6f071-kube-api-access-s4dg6\") pod \"payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp\" (UID: \"0c862e45-235b-4ee4-9efd-094389f6f071\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp" Nov 03 02:23:21.391941 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.391862 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44qgs\" (UniqueName: \"kubernetes.io/projected/f0bb7921-0ba1-4ea6-b010-504d816c2d4a-kube-api-access-44qgs\") pod \"payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv\" (UID: \"f0bb7921-0ba1-4ea6-b010-504d816c2d4a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv" Nov 03 02:23:21.392064 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.391990 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0bb7921-0ba1-4ea6-b010-504d816c2d4a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv\" (UID: \"f0bb7921-0ba1-4ea6-b010-504d816c2d4a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv" Nov 03 02:23:21.439187 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.439155 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp" Nov 03 02:23:21.500858 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.500825 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44qgs\" (UniqueName: \"kubernetes.io/projected/f0bb7921-0ba1-4ea6-b010-504d816c2d4a-kube-api-access-44qgs\") pod \"payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv\" (UID: \"f0bb7921-0ba1-4ea6-b010-504d816c2d4a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv" Nov 03 02:23:21.500987 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.500947 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0bb7921-0ba1-4ea6-b010-504d816c2d4a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv\" (UID: \"f0bb7921-0ba1-4ea6-b010-504d816c2d4a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv" Nov 03 02:23:21.502169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.502141 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0bb7921-0ba1-4ea6-b010-504d816c2d4a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv\" (UID: \"f0bb7921-0ba1-4ea6-b010-504d816c2d4a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv" Nov 03 02:23:21.558883 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.558815 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44qgs\" (UniqueName: \"kubernetes.io/projected/f0bb7921-0ba1-4ea6-b010-504d816c2d4a-kube-api-access-44qgs\") pod \"payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv\" (UID: \"f0bb7921-0ba1-4ea6-b010-504d816c2d4a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv" Nov 03 02:23:21.602186 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.602137 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv" Nov 03 02:23:21.820405 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:23:21.820372 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c862e45_235b_4ee4_9efd_094389f6f071.slice/crio-69d3c668139fec1df88e8a8518e1f6fac74dceedefc27ae7148c625dcc2bfd02 WatchSource:0}: Error finding container 69d3c668139fec1df88e8a8518e1f6fac74dceedefc27ae7148c625dcc2bfd02: Status 404 returned error can't find the container with id 69d3c668139fec1df88e8a8518e1f6fac74dceedefc27ae7148c625dcc2bfd02 Nov 03 02:23:21.821755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.821713 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp"] Nov 03 02:23:21.971037 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:23:21.970997 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bb7921_0ba1_4ea6_b010_504d816c2d4a.slice/crio-7d49ab62475f9c8fefe141d5224e56453393f9995a23e128e4f94ed38f437ed5 WatchSource:0}: Error finding container 7d49ab62475f9c8fefe141d5224e56453393f9995a23e128e4f94ed38f437ed5: Status 404 returned error can't find the container with id 7d49ab62475f9c8fefe141d5224e56453393f9995a23e128e4f94ed38f437ed5 Nov 03 02:23:21.973904 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:21.973878 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv"] Nov 03 02:23:22.616491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:22.616452 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv" event={"ID":"f0bb7921-0ba1-4ea6-b010-504d816c2d4a","Type":"ContainerStarted","Data":"5c2c20b57e9eb4a11bfe629f0db8de0740ef61170aa2818f478bab6825fb05a5"} Nov 03 02:23:22.616692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:22.616619 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv" Nov 03 02:23:22.616692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:22.616669 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv" event={"ID":"f0bb7921-0ba1-4ea6-b010-504d816c2d4a","Type":"ContainerStarted","Data":"7d49ab62475f9c8fefe141d5224e56453393f9995a23e128e4f94ed38f437ed5"} Nov 03 02:23:22.618158 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:22.618136 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp" event={"ID":"0c862e45-235b-4ee4-9efd-094389f6f071","Type":"ContainerStarted","Data":"8cc28b3d7a61a3acdaa71af608c74ad70bdee75cdd028fe378d65bd7dcab0798"} Nov 03 02:23:22.618284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:22.618176 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp" event={"ID":"0c862e45-235b-4ee4-9efd-094389f6f071","Type":"ContainerStarted","Data":"69d3c668139fec1df88e8a8518e1f6fac74dceedefc27ae7148c625dcc2bfd02"} Nov 03 02:23:22.618284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:22.618270 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp" Nov 03 02:23:22.730807 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:22.730671 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv" podStartSLOduration=1.730654822 podStartE2EDuration="1.730654822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:23:22.729679489 +0000 UTC m=+1884.413788643" watchObservedRunningTime="2026-11-03 02:23:22.730654822 +0000 UTC m=+1884.414763977" Nov 03 02:23:22.817364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:22.817331 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:22.851601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:22.851542 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp" podStartSLOduration=1.851519642 podStartE2EDuration="1.851519642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:23:22.85043325 +0000 UTC m=+1884.534542411" watchObservedRunningTime="2026-11-03 02:23:22.851519642 +0000 UTC m=+1884.535628798" Nov 03 02:23:27.492515 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:27.492480 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:28.559272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:28.559236 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:28.712379 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:28.712347 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:31.865939 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:31.865902 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd"] Nov 03 02:23:31.870333 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:31.870309 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" Nov 03 02:23:31.922010 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:31.921966 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd"] Nov 03 02:23:31.969363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:31.969335 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d63a25-serving-cert\"" Nov 03 02:23:32.026030 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:32.025996 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-openshift-service-ca\") pod \"maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd\" (UID: \"de3acde6-6c41-4773-a7e0-f4a7b32a6d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" Nov 03 02:23:32.026207 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:32.026061 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxlrv\" (UniqueName: \"kubernetes.io/projected/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-kube-api-access-fxlrv\") pod \"maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd\" (UID: \"de3acde6-6c41-4773-a7e0-f4a7b32a6d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" Nov 03 02:23:32.026329 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:32.026299 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-maas-api-tls\") pod \"maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd\" (UID: \"de3acde6-6c41-4773-a7e0-f4a7b32a6d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" Nov 03 02:23:32.026480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:32.026453 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-metrics-certs\") pod \"maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd\" (UID: \"de3acde6-6c41-4773-a7e0-f4a7b32a6d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" Nov 03 02:23:32.077218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:32.077173 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:32.128432 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:32.128352 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-maas-api-tls\") pod \"maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd\" (UID: \"de3acde6-6c41-4773-a7e0-f4a7b32a6d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" Nov 03 02:23:32.128594 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:32.128446 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-metrics-certs\") pod \"maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd\" (UID: \"de3acde6-6c41-4773-a7e0-f4a7b32a6d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" Nov 03 02:23:32.128594 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:32.128530 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-openshift-service-ca\") pod \"maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd\" (UID: \"de3acde6-6c41-4773-a7e0-f4a7b32a6d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" Nov 03 02:23:32.128594 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:32.128562 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxlrv\" (UniqueName: \"kubernetes.io/projected/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-kube-api-access-fxlrv\") pod \"maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd\" (UID: \"de3acde6-6c41-4773-a7e0-f4a7b32a6d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" Nov 03 02:23:32.129179 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:32.129154 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-openshift-service-ca\") pod \"maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd\" (UID: \"de3acde6-6c41-4773-a7e0-f4a7b32a6d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" Nov 03 02:23:32.130882 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:32.130858 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-metrics-certs\") pod \"maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd\" (UID: \"de3acde6-6c41-4773-a7e0-f4a7b32a6d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" Nov 03 02:23:32.131023 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:32.131000 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-maas-api-tls\") pod \"maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd\" (UID: \"de3acde6-6c41-4773-a7e0-f4a7b32a6d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" Nov 03 02:23:32.185458 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:32.185428 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxlrv\" (UniqueName: \"kubernetes.io/projected/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-kube-api-access-fxlrv\") pod \"maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd\" (UID: \"de3acde6-6c41-4773-a7e0-f4a7b32a6d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" Nov 03 02:23:32.187299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:32.187270 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" Nov 03 02:23:32.225111 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:32.225082 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:32.372149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:32.372117 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:32.556074 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:23:32.556042 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3acde6_6c41_4773_a7e0_f4a7b32a6d66.slice/crio-4a4c94c2364df25e42625e9a5c32ae0eb679b6029d49fbadeede9dcfb7f48263 WatchSource:0}: Error finding container 4a4c94c2364df25e42625e9a5c32ae0eb679b6029d49fbadeede9dcfb7f48263: Status 404 returned error can't find the container with id 4a4c94c2364df25e42625e9a5c32ae0eb679b6029d49fbadeede9dcfb7f48263 Nov 03 02:23:32.557604 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:32.557586 5463 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 02:23:32.558704 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:32.558681 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd"] Nov 03 02:23:32.689278 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:32.689247 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" event={"ID":"de3acde6-6c41-4773-a7e0-f4a7b32a6d66","Type":"ContainerStarted","Data":"4a4c94c2364df25e42625e9a5c32ae0eb679b6029d49fbadeede9dcfb7f48263"} Nov 03 02:23:33.628655 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:33.628629 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp" Nov 03 02:23:33.629091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:33.628702 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv" Nov 03 02:23:34.705830 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:34.705736 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" event={"ID":"de3acde6-6c41-4773-a7e0-f4a7b32a6d66","Type":"ContainerStarted","Data":"2391079a1580d9aa082895151664202fc36330cbb35a25246c5a54c44fc8b634"} Nov 03 02:23:34.705830 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:34.705811 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" Nov 03 02:23:34.821715 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:34.821668 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" podStartSLOduration=1.965626884 podStartE2EDuration="3.821653115s" totalImagesPullingTime="1.856026231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:23:34.820631411 +0000 UTC m=+1896.504740567" watchObservedRunningTime="2026-11-03 02:23:34.821653115 +0000 UTC m=+1896.505762271" Nov 03 02:23:36.961526 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:36.961494 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:40.719441 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:40.719358 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" Nov 03 02:23:41.739287 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:41.739256 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:43.007827 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:23:43.007793 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:23:45.151953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:45.151914 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:45.651798 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:45.651765 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2"] Nov 03 02:23:45.761829 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:45.761790 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk"] Nov 03 02:23:46.028594 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.028513 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:46.186514 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.186451 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:46.554061 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.554023 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2"] Nov 03 02:23:46.558177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.558152 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.607562 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.607510 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-workload-socket\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.607732 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.607574 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-credential-socket\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.607732 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.607601 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-data\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.607732 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.607669 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zw94\" (UniqueName: \"kubernetes.io/projected/17c81363-4f1f-4550-ba15-35f9e86eccf7-kube-api-access-5zw94\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.607732 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.607727 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-workload-certs\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.607986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.607770 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c81363-4f1f-4550-ba15-35f9e86eccf7-istiod-ca-cert\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.607986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.607811 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-token\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.607986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.607857 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-envoy\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.607986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.607883 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-podinfo\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.630409 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.630381 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8857cb-openshift-default-dockercfg-7tgz5\"" Nov 03 02:23:46.651232 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.650985 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2"] Nov 03 02:23:46.660036 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.660001 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:46.721214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.713528 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c81363-4f1f-4550-ba15-35f9e86eccf7-istiod-ca-cert\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.721504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.721477 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-token\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.721689 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.721669 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-envoy\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.737052 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.721800 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-podinfo\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.737428 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.737402 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-workload-socket\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.739161 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.739136 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-credential-socket\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.739389 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.739370 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-data\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.739527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.739496 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-credential-socket\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.739611 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.738971 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-workload-socket\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.739707 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.720052 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c81363-4f1f-4550-ba15-35f9e86eccf7-istiod-ca-cert\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.739817 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.739774 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-data\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.739931 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.739888 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zw94\" (UniqueName: \"kubernetes.io/projected/17c81363-4f1f-4550-ba15-35f9e86eccf7-kube-api-access-5zw94\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.740137 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.740121 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-workload-certs\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.740637 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.740616 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-workload-certs\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.775215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.774283 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-podinfo\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.776439 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.776028 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-envoy\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.811798 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.811683 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-token\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.812519 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.812489 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zw94\" (UniqueName: \"kubernetes.io/projected/17c81363-4f1f-4550-ba15-35f9e86eccf7-kube-api-access-5zw94\") pod \"e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:46.859000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.858970 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:46.872175 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:46.872149 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:47.009705 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:47.009644 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:47.320218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:47.319656 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2"] Nov 03 02:23:47.334756 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:23:47.324815 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c81363_4f1f_4550_ba15_35f9e86eccf7.slice/crio-e843ed797c815fe597795684f2a2b5c6594c4f257527aef0707777905b7fa834 WatchSource:0}: Error finding container e843ed797c815fe597795684f2a2b5c6594c4f257527aef0707777905b7fa834: Status 404 returned error can't find the container with id e843ed797c815fe597795684f2a2b5c6594c4f257527aef0707777905b7fa834 Nov 03 02:23:47.334756 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:47.328487 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:23:47.334756 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:47.328645 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:23:47.334756 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:47.328727 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:23:47.808720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:47.808039 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" event={"ID":"17c81363-4f1f-4550-ba15-35f9e86eccf7","Type":"ContainerStarted","Data":"84400737403c98c63902b2439174cd25ae46e5336a60892423e2b597ebf71be0"} Nov 03 02:23:47.808720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:47.808135 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" event={"ID":"17c81363-4f1f-4550-ba15-35f9e86eccf7","Type":"ContainerStarted","Data":"e843ed797c815fe597795684f2a2b5c6594c4f257527aef0707777905b7fa834"} Nov 03 02:23:47.873149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:47.873121 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:47.925155 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:47.925100 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" podStartSLOduration=1.9250870519999999 podStartE2EDuration="1.925087052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:23:47.923592096 +0000 UTC m=+1909.607701252" watchObservedRunningTime="2026-11-03 02:23:47.925087052 +0000 UTC m=+1909.609196207" Nov 03 02:23:48.878104 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:48.878070 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:49.823709 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:49.823680 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:49.824598 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:49.824576 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:23:50.652114 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:50.652066 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2" podUID="0d84ec5e-7992-48e5-83ab-816302748e41" containerName="payload-processing" containerID="cri-o://2abeb84d7418a307cf4751d8e2dcf3a956e17d3e878c9f404282d0347837074c" gracePeriod=25 Nov 03 02:23:50.762304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:50.762241 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk" podUID="ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa" containerName="payload-pre-processing" containerID="cri-o://603b32a12196d3e59b2fde25663b2b1efa65b727fd9a3bb3a4051e04b5d682ff" gracePeriod=25 Nov 03 02:23:50.832800 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:50.832768 5463 generic.go:356] "Generic (PLEG): container finished" podID="0d84ec5e-7992-48e5-83ab-816302748e41" containerID="2abeb84d7418a307cf4751d8e2dcf3a956e17d3e878c9f404282d0347837074c" exitCode=0 Nov 03 02:23:50.832967 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:50.832847 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2" event={"ID":"0d84ec5e-7992-48e5-83ab-816302748e41","Type":"ContainerDied","Data":"2abeb84d7418a307cf4751d8e2dcf3a956e17d3e878c9f404282d0347837074c"} Nov 03 02:23:50.954473 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:50.949966 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2" Nov 03 02:23:51.040893 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.040869 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk" Nov 03 02:23:51.071785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.071758 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:51.099604 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.099573 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d84ec5e-7992-48e5-83ab-816302748e41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d84ec5e-7992-48e5-83ab-816302748e41-plugins-config-volume\") pod \"0d84ec5e-7992-48e5-83ab-816302748e41\" (UID: \"0d84ec5e-7992-48e5-83ab-816302748e41\") " Nov 03 02:23:51.099763 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.099647 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d84ec5e-7992-48e5-83ab-816302748e41-kube-api-access-zs5c4\" (UniqueName: \"kubernetes.io/projected/0d84ec5e-7992-48e5-83ab-816302748e41-kube-api-access-zs5c4\") pod \"0d84ec5e-7992-48e5-83ab-816302748e41\" (UID: \"0d84ec5e-7992-48e5-83ab-816302748e41\") " Nov 03 02:23:51.099995 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.099971 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d84ec5e-7992-48e5-83ab-816302748e41-plugins-config-volume" pod "0d84ec5e-7992-48e5-83ab-816302748e41" (UID: "0d84ec5e-7992-48e5-83ab-816302748e41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:23:51.100262 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.100244 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d84ec5e-7992-48e5-83ab-816302748e41-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:51.101703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.101679 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d84ec5e-7992-48e5-83ab-816302748e41-kube-api-access-zs5c4" pod "0d84ec5e-7992-48e5-83ab-816302748e41" (UID: "0d84ec5e-7992-48e5-83ab-816302748e41"). InnerVolumeSpecName "kube-api-access-zs5c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:23:51.202776 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.202672 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa-kube-api-access-q7j2l\" (UniqueName: \"kubernetes.io/projected/ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa-kube-api-access-q7j2l\") pod \"ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa\" (UID: \"ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa\") " Nov 03 02:23:51.202776 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.202748 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa-plugins-config-volume\") pod \"ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa\" (UID: \"ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa\") " Nov 03 02:23:51.203310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.203279 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa-plugins-config-volume" pod "ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa" (UID: "ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:23:51.203430 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.203345 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs5c4\" (UniqueName: \"kubernetes.io/projected/0d84ec5e-7992-48e5-83ab-816302748e41-kube-api-access-zs5c4\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:51.205478 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.205454 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa-kube-api-access-q7j2l" pod "ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa" (UID: "ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa"). InnerVolumeSpecName "kube-api-access-q7j2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:23:51.244316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.244288 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:51.303991 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.303968 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7j2l\" (UniqueName: \"kubernetes.io/projected/ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa-kube-api-access-q7j2l\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:51.303991 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.303990 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:51.498335 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.498244 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:51.842479 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.842401 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2" Nov 03 02:23:51.842922 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.842410 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2" event={"ID":"0d84ec5e-7992-48e5-83ab-816302748e41","Type":"ContainerDied","Data":"e5b05eac985330d33ad2f32338eddfbf9dda83568720d7ab7fe47594bcbbd970"} Nov 03 02:23:51.842922 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.842548 5463 scope.go:122] "RemoveContainer" containerID="2abeb84d7418a307cf4751d8e2dcf3a956e17d3e878c9f404282d0347837074c" Nov 03 02:23:51.843983 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.843962 5463 generic.go:356] "Generic (PLEG): container finished" podID="ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa" containerID="603b32a12196d3e59b2fde25663b2b1efa65b727fd9a3bb3a4051e04b5d682ff" exitCode=0 Nov 03 02:23:51.844075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.844021 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk" Nov 03 02:23:51.844075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.844047 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk" event={"ID":"ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa","Type":"ContainerDied","Data":"603b32a12196d3e59b2fde25663b2b1efa65b727fd9a3bb3a4051e04b5d682ff"} Nov 03 02:23:51.844230 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.844094 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk" event={"ID":"ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa","Type":"ContainerDied","Data":"12f274e25a77b6306e5a1a7dc96a70593d736970a014a5f9a363ae581b39e522"} Nov 03 02:23:51.858795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.858774 5463 scope.go:122] "RemoveContainer" containerID="603b32a12196d3e59b2fde25663b2b1efa65b727fd9a3bb3a4051e04b5d682ff" Nov 03 02:23:51.868297 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.868278 5463 scope.go:122] "RemoveContainer" containerID="603b32a12196d3e59b2fde25663b2b1efa65b727fd9a3bb3a4051e04b5d682ff" Nov 03 02:23:51.868600 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:23:51.868571 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"603b32a12196d3e59b2fde25663b2b1efa65b727fd9a3bb3a4051e04b5d682ff\": container with ID starting with 603b32a12196d3e59b2fde25663b2b1efa65b727fd9a3bb3a4051e04b5d682ff not found: ID does not exist" containerID="603b32a12196d3e59b2fde25663b2b1efa65b727fd9a3bb3a4051e04b5d682ff" Nov 03 02:23:51.868702 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:51.868608 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"603b32a12196d3e59b2fde25663b2b1efa65b727fd9a3bb3a4051e04b5d682ff"} err="failed to get container status \"603b32a12196d3e59b2fde25663b2b1efa65b727fd9a3bb3a4051e04b5d682ff\": rpc error: code = NotFound desc = could not find container \"603b32a12196d3e59b2fde25663b2b1efa65b727fd9a3bb3a4051e04b5d682ff\": container with ID starting with 603b32a12196d3e59b2fde25663b2b1efa65b727fd9a3bb3a4051e04b5d682ff not found: ID does not exist" Nov 03 02:23:52.008815 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:52.008779 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2"] Nov 03 02:23:52.057602 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:52.057565 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-56299c-65f8c74c57-6nww2"] Nov 03 02:23:52.120780 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:52.120707 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk"] Nov 03 02:23:52.123512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:52.123490 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-555656b99-5swbk"] Nov 03 02:23:52.445030 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:52.445001 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d84ec5e-7992-48e5-83ab-816302748e41" path="/var/lib/kubelet/pods/0d84ec5e-7992-48e5-83ab-816302748e41/volumes" Nov 03 02:23:52.445456 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:52.445441 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa" path="/var/lib/kubelet/pods/ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa/volumes" Nov 03 02:23:54.016847 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.016811 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:54.248322 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.248283 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:54.438121 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.438089 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:54.646206 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.646162 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc"] Nov 03 02:23:54.647123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.647097 5463 cpu_manager.go:414] "containerMap: removing container" podUID="0d84ec5e-7992-48e5-83ab-816302748e41" containerName="payload-processing" Nov 03 02:23:54.647123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.647122 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d84ec5e-7992-48e5-83ab-816302748e41" containerName="payload-processing" Nov 03 02:23:54.647380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.647204 5463 cpu_manager.go:414] "containerMap: removing container" podUID="ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa" containerName="payload-pre-processing" Nov 03 02:23:54.647380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.647213 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa" containerName="payload-pre-processing" Nov 03 02:23:54.647380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.647360 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1a04a5-b15b-4092-aaf3-4c3f7b908ffa" containerName="payload-pre-processing" Nov 03 02:23:54.647380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.647379 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d84ec5e-7992-48e5-83ab-816302748e41" containerName="payload-processing" Nov 03 02:23:54.651459 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.651434 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc" Nov 03 02:23:54.735269 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.714693 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc"] Nov 03 02:23:54.753995 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.753956 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8857cb-dockercfg-q9z5r\"" Nov 03 02:23:54.754315 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.754297 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8857cb\"" Nov 03 02:23:54.755144 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.755116 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c9cf2a-a41a-45c3-897e-140344ea67ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc\" (UID: \"b7c9cf2a-a41a-45c3-897e-140344ea67ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc" Nov 03 02:23:54.756453 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.756428 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjffp\" (UniqueName: \"kubernetes.io/projected/b7c9cf2a-a41a-45c3-897e-140344ea67ff-kube-api-access-pjffp\") pod \"payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc\" (UID: \"b7c9cf2a-a41a-45c3-897e-140344ea67ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc" Nov 03 02:23:54.858392 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.858282 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjffp\" (UniqueName: \"kubernetes.io/projected/b7c9cf2a-a41a-45c3-897e-140344ea67ff-kube-api-access-pjffp\") pod \"payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc\" (UID: \"b7c9cf2a-a41a-45c3-897e-140344ea67ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc" Nov 03 02:23:54.859219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.858604 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c9cf2a-a41a-45c3-897e-140344ea67ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc\" (UID: \"b7c9cf2a-a41a-45c3-897e-140344ea67ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc" Nov 03 02:23:54.860579 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.859901 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c9cf2a-a41a-45c3-897e-140344ea67ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc\" (UID: \"b7c9cf2a-a41a-45c3-897e-140344ea67ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc" Nov 03 02:23:54.879560 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.879536 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq"] Nov 03 02:23:54.884277 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.884258 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq" Nov 03 02:23:54.884459 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.884436 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq"] Nov 03 02:23:54.917794 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.917769 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjffp\" (UniqueName: \"kubernetes.io/projected/b7c9cf2a-a41a-45c3-897e-140344ea67ff-kube-api-access-pjffp\") pod \"payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc\" (UID: \"b7c9cf2a-a41a-45c3-897e-140344ea67ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc" Nov 03 02:23:54.959932 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.959895 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b8b53d3-cac3-4296-a612-e4f4604ce041-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq\" (UID: \"2b8b53d3-cac3-4296-a612-e4f4604ce041\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq" Nov 03 02:23:54.960091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.960004 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck68n\" (UniqueName: \"kubernetes.io/projected/2b8b53d3-cac3-4296-a612-e4f4604ce041-kube-api-access-ck68n\") pod \"payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq\" (UID: \"2b8b53d3-cac3-4296-a612-e4f4604ce041\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq" Nov 03 02:23:54.969760 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:54.969698 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc" Nov 03 02:23:55.067980 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:55.066048 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b8b53d3-cac3-4296-a612-e4f4604ce041-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq\" (UID: \"2b8b53d3-cac3-4296-a612-e4f4604ce041\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq" Nov 03 02:23:55.067980 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:55.066163 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck68n\" (UniqueName: \"kubernetes.io/projected/2b8b53d3-cac3-4296-a612-e4f4604ce041-kube-api-access-ck68n\") pod \"payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq\" (UID: \"2b8b53d3-cac3-4296-a612-e4f4604ce041\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq" Nov 03 02:23:55.068461 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:55.068398 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b8b53d3-cac3-4296-a612-e4f4604ce041-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq\" (UID: \"2b8b53d3-cac3-4296-a612-e4f4604ce041\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq" Nov 03 02:23:55.130271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:55.129964 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck68n\" (UniqueName: \"kubernetes.io/projected/2b8b53d3-cac3-4296-a612-e4f4604ce041-kube-api-access-ck68n\") pod \"payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq\" (UID: \"2b8b53d3-cac3-4296-a612-e4f4604ce041\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq" Nov 03 02:23:55.203601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:55.203131 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq" Nov 03 02:23:55.363282 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:23:55.363183 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c9cf2a_a41a_45c3_897e_140344ea67ff.slice/crio-187565bf9bb2bc0cd1971158da4f6e2fde9078ec5b1b1870f89d5b2babdd894f WatchSource:0}: Error finding container 187565bf9bb2bc0cd1971158da4f6e2fde9078ec5b1b1870f89d5b2babdd894f: Status 404 returned error can't find the container with id 187565bf9bb2bc0cd1971158da4f6e2fde9078ec5b1b1870f89d5b2babdd894f Nov 03 02:23:55.365327 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:55.365125 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc"] Nov 03 02:23:55.598995 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:23:55.598953 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b8b53d3_cac3_4296_a612_e4f4604ce041.slice/crio-4ea416eb4f069b2b7920cfa5cfe61f73c86803c676b1be4d878b87abc158d850 WatchSource:0}: Error finding container 4ea416eb4f069b2b7920cfa5cfe61f73c86803c676b1be4d878b87abc158d850: Status 404 returned error can't find the container with id 4ea416eb4f069b2b7920cfa5cfe61f73c86803c676b1be4d878b87abc158d850 Nov 03 02:23:55.601081 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:55.601054 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq"] Nov 03 02:23:55.885696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:55.885591 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc" event={"ID":"b7c9cf2a-a41a-45c3-897e-140344ea67ff","Type":"ContainerStarted","Data":"ec5271a67eb644d44b7dde24a129d279130d039d0d2b7b35674916ae3abc4ec1"} Nov 03 02:23:55.885696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:55.885660 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc" Nov 03 02:23:55.885696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:55.885682 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc" event={"ID":"b7c9cf2a-a41a-45c3-897e-140344ea67ff","Type":"ContainerStarted","Data":"187565bf9bb2bc0cd1971158da4f6e2fde9078ec5b1b1870f89d5b2babdd894f"} Nov 03 02:23:55.887350 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:55.887317 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq" event={"ID":"2b8b53d3-cac3-4296-a612-e4f4604ce041","Type":"ContainerStarted","Data":"31b46eb25f4d08c2f31864aedc135b09b8e04ea16f0732dfe287dbbcf94d02d6"} Nov 03 02:23:55.887492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:55.887366 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq" event={"ID":"2b8b53d3-cac3-4296-a612-e4f4604ce041","Type":"ContainerStarted","Data":"4ea416eb4f069b2b7920cfa5cfe61f73c86803c676b1be4d878b87abc158d850"} Nov 03 02:23:55.887492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:55.887410 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq" Nov 03 02:23:56.000933 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:56.000879 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc" podStartSLOduration=2.000864732 podStartE2EDuration="2.000864732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:23:55.9997545 +0000 UTC m=+1917.683863667" watchObservedRunningTime="2026-11-03 02:23:56.000864732 +0000 UTC m=+1917.684973887" Nov 03 02:23:56.115223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:23:56.115160 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq" podStartSLOduration=2.115144568 podStartE2EDuration="2.115144568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:23:56.112290313 +0000 UTC m=+1917.796399471" watchObservedRunningTime="2026-11-03 02:23:56.115144568 +0000 UTC m=+1917.799253722" Nov 03 02:24:01.084584 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:01.084544 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:01.235485 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:01.235453 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:04.288066 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:04.288035 5463 scope.go:122] "RemoveContainer" containerID="b33e9dbef23ec658584cffdb36dc809c9b9bd6d60d597b994763896cbe974308" Nov 03 02:24:04.636866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:04.636786 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:05.109282 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:05.109254 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:06.196846 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:06.196805 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75"] Nov 03 02:24:06.202520 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:06.202493 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75" Nov 03 02:24:06.252216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:06.252172 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75"] Nov 03 02:24:06.269041 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:06.269006 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e20d8b1-8b29-4d61-a854-071821501552-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75\" (UID: \"2e20d8b1-8b29-4d61-a854-071821501552\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75" Nov 03 02:24:06.269215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:06.269164 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xwlv\" (UniqueName: \"kubernetes.io/projected/2e20d8b1-8b29-4d61-a854-071821501552-kube-api-access-8xwlv\") pod \"payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75\" (UID: \"2e20d8b1-8b29-4d61-a854-071821501552\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75" Nov 03 02:24:06.298549 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:06.298503 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9bd3ab\"" Nov 03 02:24:06.298838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:06.298817 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9bd3ab-dockercfg-q5ppp\"" Nov 03 02:24:06.370527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:06.370488 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e20d8b1-8b29-4d61-a854-071821501552-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75\" (UID: \"2e20d8b1-8b29-4d61-a854-071821501552\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75" Nov 03 02:24:06.370692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:06.370597 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xwlv\" (UniqueName: \"kubernetes.io/projected/2e20d8b1-8b29-4d61-a854-071821501552-kube-api-access-8xwlv\") pod \"payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75\" (UID: \"2e20d8b1-8b29-4d61-a854-071821501552\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75" Nov 03 02:24:06.371474 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:06.371446 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e20d8b1-8b29-4d61-a854-071821501552-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75\" (UID: \"2e20d8b1-8b29-4d61-a854-071821501552\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75" Nov 03 02:24:06.427548 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:06.427515 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xwlv\" (UniqueName: \"kubernetes.io/projected/2e20d8b1-8b29-4d61-a854-071821501552-kube-api-access-8xwlv\") pod \"payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75\" (UID: \"2e20d8b1-8b29-4d61-a854-071821501552\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75" Nov 03 02:24:06.519487 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:06.519417 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75" Nov 03 02:24:06.899504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:06.899471 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq" Nov 03 02:24:06.899689 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:06.899674 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc" Nov 03 02:24:06.906580 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:24:06.906553 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e20d8b1_8b29_4d61_a854_071821501552.slice/crio-ac99dd851886be7779b0ac8f068fb3722288d5476ac3bdb63f2ff159b3f3e26b WatchSource:0}: Error finding container ac99dd851886be7779b0ac8f068fb3722288d5476ac3bdb63f2ff159b3f3e26b: Status 404 returned error can't find the container with id ac99dd851886be7779b0ac8f068fb3722288d5476ac3bdb63f2ff159b3f3e26b Nov 03 02:24:06.909643 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:06.909619 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75"] Nov 03 02:24:07.004284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:07.004254 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659"] Nov 03 02:24:07.007698 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:07.007676 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659" Nov 03 02:24:07.013637 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:07.013609 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75" event={"ID":"2e20d8b1-8b29-4d61-a854-071821501552","Type":"ContainerStarted","Data":"ac99dd851886be7779b0ac8f068fb3722288d5476ac3bdb63f2ff159b3f3e26b"} Nov 03 02:24:07.092486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:07.092438 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659"] Nov 03 02:24:07.096457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:07.094036 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98497121-d021-473e-83f8-fe5e3c3101ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659\" (UID: \"98497121-d021-473e-83f8-fe5e3c3101ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659" Nov 03 02:24:07.096457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:07.094239 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvbkq\" (UniqueName: \"kubernetes.io/projected/98497121-d021-473e-83f8-fe5e3c3101ac-kube-api-access-xvbkq\") pod \"payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659\" (UID: \"98497121-d021-473e-83f8-fe5e3c3101ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659" Nov 03 02:24:07.195725 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:07.195630 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvbkq\" (UniqueName: \"kubernetes.io/projected/98497121-d021-473e-83f8-fe5e3c3101ac-kube-api-access-xvbkq\") pod \"payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659\" (UID: \"98497121-d021-473e-83f8-fe5e3c3101ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659" Nov 03 02:24:07.195912 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:07.195790 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98497121-d021-473e-83f8-fe5e3c3101ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659\" (UID: \"98497121-d021-473e-83f8-fe5e3c3101ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659" Nov 03 02:24:07.196799 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:07.196776 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98497121-d021-473e-83f8-fe5e3c3101ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659\" (UID: \"98497121-d021-473e-83f8-fe5e3c3101ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659" Nov 03 02:24:07.253483 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:07.253442 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvbkq\" (UniqueName: \"kubernetes.io/projected/98497121-d021-473e-83f8-fe5e3c3101ac-kube-api-access-xvbkq\") pod \"payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659\" (UID: \"98497121-d021-473e-83f8-fe5e3c3101ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659" Nov 03 02:24:07.325681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:07.325648 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659" Nov 03 02:24:07.716614 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:24:07.716582 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98497121_d021_473e_83f8_fe5e3c3101ac.slice/crio-a0d385c3f7f228d0552117a06b1c07c5fea25e97a4dc9ded6fdd48868fc0cbfa WatchSource:0}: Error finding container a0d385c3f7f228d0552117a06b1c07c5fea25e97a4dc9ded6fdd48868fc0cbfa: Status 404 returned error can't find the container with id a0d385c3f7f228d0552117a06b1c07c5fea25e97a4dc9ded6fdd48868fc0cbfa Nov 03 02:24:07.723245 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:07.720101 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659"] Nov 03 02:24:07.746484 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:07.746457 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:08.021780 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:08.021678 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659" event={"ID":"98497121-d021-473e-83f8-fe5e3c3101ac","Type":"ContainerStarted","Data":"cbc0efd7ecd027d355ead1f383dd1c01d407185ec94810a1b13541174f5d8172"} Nov 03 02:24:08.021780 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:08.021748 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659" event={"ID":"98497121-d021-473e-83f8-fe5e3c3101ac","Type":"ContainerStarted","Data":"a0d385c3f7f228d0552117a06b1c07c5fea25e97a4dc9ded6fdd48868fc0cbfa"} Nov 03 02:24:08.021780 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:08.021780 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659" Nov 03 02:24:08.023809 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:08.023782 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75" event={"ID":"2e20d8b1-8b29-4d61-a854-071821501552","Type":"ContainerStarted","Data":"54f6d5fb3064bce8e875315a97ae9a45b96063b630a97e3f04203a0551bd03f5"} Nov 03 02:24:08.023953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:08.023831 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75" Nov 03 02:24:08.136641 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:08.136580 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659" podStartSLOduration=2.1365613039999998 podStartE2EDuration="2.136561304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:24:08.135357251 +0000 UTC m=+1929.819466406" watchObservedRunningTime="2026-11-03 02:24:08.136561304 +0000 UTC m=+1929.820670462" Nov 03 02:24:08.306838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:08.304547 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75" podStartSLOduration=2.304510423 podStartE2EDuration="2.304510423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:24:08.262791217 +0000 UTC m=+1929.946900375" watchObservedRunningTime="2026-11-03 02:24:08.304510423 +0000 UTC m=+1929.988619580" Nov 03 02:24:09.366153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:09.364928 5463 patch_prober.go:29] interesting pod/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": read tcp 10.133.0.2:53694->10.133.0.174:15021: read: connection reset by peer" start-of-body= Nov 03 02:24:09.366153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:09.364985 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" podUID="b2a15ca1-f5c6-41af-849f-9e727badc8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": read tcp 10.133.0.2:53694->10.133.0.174:15021: read: connection reset by peer" Nov 03 02:24:10.040657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:10.040624 5463 generic.go:356] "Generic (PLEG): container finished" podID="b2a15ca1-f5c6-41af-849f-9e727badc8a7" containerID="53792e0b9d91a6c631fdd5b4462e467a020b9048d53f72f129dd7dd57e8ab792" exitCode=0 Nov 03 02:24:10.040843 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:10.040699 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" event={"ID":"b2a15ca1-f5c6-41af-849f-9e727badc8a7","Type":"ContainerDied","Data":"53792e0b9d91a6c631fdd5b4462e467a020b9048d53f72f129dd7dd57e8ab792"} Nov 03 02:24:10.041080 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:10.041065 5463 scope.go:122] "RemoveContainer" containerID="53792e0b9d91a6c631fdd5b4462e467a020b9048d53f72f129dd7dd57e8ab792" Nov 03 02:24:10.042490 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:10.042461 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:24:10.042566 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:10.042536 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:24:10.042604 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:10.042571 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:24:11.058215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:11.051957 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" event={"ID":"b2a15ca1-f5c6-41af-849f-9e727badc8a7","Type":"ContainerStarted","Data":"8ce30abaaf6a34be9771565169b2a6619dba3633f41309fdbe31766297ff0168"} Nov 03 02:24:11.774299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:11.774245 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:24:11.774630 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:11.774534 5463 patch_prober.go:29] interesting pod/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 03 02:24:11.774630 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:11.774579 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" podUID="b2a15ca1-f5c6-41af-849f-9e727badc8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 03 02:24:12.695623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:12.695583 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:12.774730 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:12.774689 5463 patch_prober.go:29] interesting pod/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 03 02:24:12.774944 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:12.774753 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" podUID="b2a15ca1-f5c6-41af-849f-9e727badc8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 03 02:24:13.007935 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:24:13.007847 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:24:13.774519 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:13.774478 5463 patch_prober.go:29] interesting pod/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 03 02:24:13.775032 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:13.774555 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" podUID="b2a15ca1-f5c6-41af-849f-9e727badc8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 03 02:24:13.974619 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:13.974577 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:14.774597 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:14.774558 5463 patch_prober.go:29] interesting pod/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 03 02:24:14.774998 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:14.774632 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" podUID="b2a15ca1-f5c6-41af-849f-9e727badc8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 03 02:24:15.773895 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:15.773862 5463 patch_prober.go:29] interesting pod/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 03 02:24:15.774086 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:15.773913 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" podUID="b2a15ca1-f5c6-41af-849f-9e727badc8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 03 02:24:16.201567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:16.201527 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:16.679685 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:16.679653 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:16.778818 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:16.778784 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:24:16.779376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:16.779355 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:24:16.780184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:16.780162 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:24:17.768639 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:17.768607 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk"] Nov 03 02:24:17.784700 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:17.784668 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk" Nov 03 02:24:17.825886 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:17.825859 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk"] Nov 03 02:24:17.868959 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:17.868921 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5303d364-11ee-429b-8545-7a498d93213b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk\" (UID: \"5303d364-11ee-429b-8545-7a498d93213b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk" Nov 03 02:24:17.869133 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:17.869055 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwhpf\" (UniqueName: \"kubernetes.io/projected/5303d364-11ee-429b-8545-7a498d93213b-kube-api-access-mwhpf\") pod \"payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk\" (UID: \"5303d364-11ee-429b-8545-7a498d93213b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk" Nov 03 02:24:17.870967 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:17.870946 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ad4591\"" Nov 03 02:24:17.871103 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:17.870967 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ad4591-dockercfg-xb86q\"" Nov 03 02:24:17.970174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:17.970144 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5303d364-11ee-429b-8545-7a498d93213b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk\" (UID: \"5303d364-11ee-429b-8545-7a498d93213b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk" Nov 03 02:24:17.970350 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:17.970236 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwhpf\" (UniqueName: \"kubernetes.io/projected/5303d364-11ee-429b-8545-7a498d93213b-kube-api-access-mwhpf\") pod \"payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk\" (UID: \"5303d364-11ee-429b-8545-7a498d93213b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk" Nov 03 02:24:17.970907 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:17.970887 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5303d364-11ee-429b-8545-7a498d93213b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk\" (UID: \"5303d364-11ee-429b-8545-7a498d93213b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk" Nov 03 02:24:18.027730 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:18.027666 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwhpf\" (UniqueName: \"kubernetes.io/projected/5303d364-11ee-429b-8545-7a498d93213b-kube-api-access-mwhpf\") pod \"payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk\" (UID: \"5303d364-11ee-429b-8545-7a498d93213b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk" Nov 03 02:24:18.099602 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:18.099580 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk" Nov 03 02:24:18.472438 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:24:18.472410 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5303d364_11ee_429b_8545_7a498d93213b.slice/crio-0045b7f529eacb8ae1e3fcb704682e4fd3cf78177f92ea187587c2aaf0a4286e WatchSource:0}: Error finding container 0045b7f529eacb8ae1e3fcb704682e4fd3cf78177f92ea187587c2aaf0a4286e: Status 404 returned error can't find the container with id 0045b7f529eacb8ae1e3fcb704682e4fd3cf78177f92ea187587c2aaf0a4286e Nov 03 02:24:18.475011 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:18.474987 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk"] Nov 03 02:24:18.570826 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:18.570794 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p"] Nov 03 02:24:18.576574 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:18.576554 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p" Nov 03 02:24:18.633634 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:18.633599 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p"] Nov 03 02:24:18.680277 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:18.680239 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df8dk\" (UniqueName: \"kubernetes.io/projected/746d6093-1cc8-4ef2-90ce-22f2a1a5a973-kube-api-access-df8dk\") pod \"payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p\" (UID: \"746d6093-1cc8-4ef2-90ce-22f2a1a5a973\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p" Nov 03 02:24:18.680553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:18.680528 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/746d6093-1cc8-4ef2-90ce-22f2a1a5a973-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p\" (UID: \"746d6093-1cc8-4ef2-90ce-22f2a1a5a973\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p" Nov 03 02:24:18.781334 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:18.781244 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/746d6093-1cc8-4ef2-90ce-22f2a1a5a973-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p\" (UID: \"746d6093-1cc8-4ef2-90ce-22f2a1a5a973\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p" Nov 03 02:24:18.781798 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:18.781339 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df8dk\" (UniqueName: \"kubernetes.io/projected/746d6093-1cc8-4ef2-90ce-22f2a1a5a973-kube-api-access-df8dk\") pod \"payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p\" (UID: \"746d6093-1cc8-4ef2-90ce-22f2a1a5a973\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p" Nov 03 02:24:18.781979 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:18.781956 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/746d6093-1cc8-4ef2-90ce-22f2a1a5a973-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p\" (UID: \"746d6093-1cc8-4ef2-90ce-22f2a1a5a973\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p" Nov 03 02:24:18.838223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:18.838159 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df8dk\" (UniqueName: \"kubernetes.io/projected/746d6093-1cc8-4ef2-90ce-22f2a1a5a973-kube-api-access-df8dk\") pod \"payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p\" (UID: \"746d6093-1cc8-4ef2-90ce-22f2a1a5a973\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p" Nov 03 02:24:18.892398 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:18.892365 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p" Nov 03 02:24:19.034184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:19.034084 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659" Nov 03 02:24:19.034583 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:19.034561 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75" Nov 03 02:24:19.142174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:19.142133 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk" event={"ID":"5303d364-11ee-429b-8545-7a498d93213b","Type":"ContainerStarted","Data":"0564c5ed614a5552c58d26f4ada49c5e74b2c5480fe3304d0099e61ab6dacb5d"} Nov 03 02:24:19.142330 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:19.142209 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk" event={"ID":"5303d364-11ee-429b-8545-7a498d93213b","Type":"ContainerStarted","Data":"0045b7f529eacb8ae1e3fcb704682e4fd3cf78177f92ea187587c2aaf0a4286e"} Nov 03 02:24:19.142330 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:19.142247 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk" Nov 03 02:24:19.228955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:19.228925 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:19.295248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:19.295108 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p"] Nov 03 02:24:19.309732 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:24:19.309696 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746d6093_1cc8_4ef2_90ce_22f2a1a5a973.slice/crio-9ec1c39aaa3e95a4a477429210e286383ca8016d87a3687ba193f411e6f4c077 WatchSource:0}: Error finding container 9ec1c39aaa3e95a4a477429210e286383ca8016d87a3687ba193f411e6f4c077: Status 404 returned error can't find the container with id 9ec1c39aaa3e95a4a477429210e286383ca8016d87a3687ba193f411e6f4c077 Nov 03 02:24:20.152670 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:20.152626 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p" event={"ID":"746d6093-1cc8-4ef2-90ce-22f2a1a5a973","Type":"ContainerStarted","Data":"a16463f03837aaa2af0d93d636ff51035593a60377d09d8aabf350827dd30de8"} Nov 03 02:24:20.153153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:20.152727 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p" event={"ID":"746d6093-1cc8-4ef2-90ce-22f2a1a5a973","Type":"ContainerStarted","Data":"9ec1c39aaa3e95a4a477429210e286383ca8016d87a3687ba193f411e6f4c077"} Nov 03 02:24:20.153153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:20.152925 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p" Nov 03 02:24:20.269517 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:20.269470 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk" podStartSLOduration=3.269456214 podStartE2EDuration="3.269456214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:24:19.380373272 +0000 UTC m=+1941.064482431" watchObservedRunningTime="2026-11-03 02:24:20.269456214 +0000 UTC m=+1941.953565408" Nov 03 02:24:20.270750 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:20.270714 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p" podStartSLOduration=2.270703831 podStartE2EDuration="2.270703831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:24:20.267451757 +0000 UTC m=+1941.951560913" watchObservedRunningTime="2026-11-03 02:24:20.270703831 +0000 UTC m=+1941.954813015" Nov 03 02:24:21.115464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:21.115432 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:23.994956 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:23.994923 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:26.696048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:26.696015 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:28.770123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:28.770081 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:28.950507 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:28.950434 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:29.094146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:29.094060 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:30.155810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:30.155783 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk" Nov 03 02:24:31.166132 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:31.166103 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p" Nov 03 02:24:33.923631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:33.923577 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:36.334900 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:36.334863 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:38.590488 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:38.590440 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:41.904225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:41.904171 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:43.008566 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:24:43.008533 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:24:43.423335 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:43.423305 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:45.196502 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:45.196466 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:48.734793 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:48.734753 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:54.119752 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:54.119712 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:54.722389 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:54.722352 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:56.456937 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:56.456903 5463 generic.go:356] "Generic (PLEG): container finished" podID="e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" containerID="ec0e3cbe1669b07275fd6906e388407cd2fac1893c9dba0801903c5c8bf8f3e0" exitCode=0 Nov 03 02:24:56.457518 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:56.456982 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" event={"ID":"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f","Type":"ContainerDied","Data":"ec0e3cbe1669b07275fd6906e388407cd2fac1893c9dba0801903c5c8bf8f3e0"} Nov 03 02:24:56.457518 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:56.457474 5463 scope.go:122] "RemoveContainer" containerID="ec0e3cbe1669b07275fd6906e388407cd2fac1893c9dba0801903c5c8bf8f3e0" Nov 03 02:24:56.459232 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:56.459169 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:24:56.459363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:56.459271 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:24:56.459363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:56.459304 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:24:57.472682 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:57.472641 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:57.479736 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:57.479699 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" event={"ID":"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f","Type":"ContainerStarted","Data":"5e009602cb886f697076313344b625625b187ca3d2faea1752a2be34da99a916"} Nov 03 02:24:57.905463 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:57.905430 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:58.305393 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:58.305311 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:24:58.306363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:58.306324 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 02:24:58.306516 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:58.306393 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" podUID="e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 02:24:58.379290 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:58.379257 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:59.304393 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:59.304359 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 02:24:59.304870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:59.304418 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" podUID="e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 02:24:59.475266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:59.475230 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9"] Nov 03 02:24:59.483449 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:59.483411 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9" Nov 03 02:24:59.520253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:59.520224 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9"] Nov 03 02:24:59.578624 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:59.578528 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-56299c\"" Nov 03 02:24:59.578865 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:59.578527 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-56299c-dockercfg-zt26m\"" Nov 03 02:24:59.580849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:59.580824 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlb5b\" (UniqueName: \"kubernetes.io/projected/c97f240a-07b4-4e32-be49-4b7a9f077012-kube-api-access-tlb5b\") pod \"payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9\" (UID: \"c97f240a-07b4-4e32-be49-4b7a9f077012\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9" Nov 03 02:24:59.580991 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:59.580929 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97f240a-07b4-4e32-be49-4b7a9f077012-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9\" (UID: \"c97f240a-07b4-4e32-be49-4b7a9f077012\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9" Nov 03 02:24:59.681979 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:59.681939 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97f240a-07b4-4e32-be49-4b7a9f077012-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9\" (UID: \"c97f240a-07b4-4e32-be49-4b7a9f077012\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9" Nov 03 02:24:59.682176 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:59.682152 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlb5b\" (UniqueName: \"kubernetes.io/projected/c97f240a-07b4-4e32-be49-4b7a9f077012-kube-api-access-tlb5b\") pod \"payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9\" (UID: \"c97f240a-07b4-4e32-be49-4b7a9f077012\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9" Nov 03 02:24:59.682775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:59.682754 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97f240a-07b4-4e32-be49-4b7a9f077012-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9\" (UID: \"c97f240a-07b4-4e32-be49-4b7a9f077012\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9" Nov 03 02:24:59.738944 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:59.738915 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlb5b\" (UniqueName: \"kubernetes.io/projected/c97f240a-07b4-4e32-be49-4b7a9f077012-kube-api-access-tlb5b\") pod \"payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9\" (UID: \"c97f240a-07b4-4e32-be49-4b7a9f077012\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9" Nov 03 02:24:59.799871 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:24:59.799839 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9" Nov 03 02:25:00.196256 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:25:00.196221 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97f240a_07b4_4e32_be49_4b7a9f077012.slice/crio-b49229f7455d7c132291c423b45ef188ced27cdd5afe2f2fe78ec4a0bd449b79 WatchSource:0}: Error finding container b49229f7455d7c132291c423b45ef188ced27cdd5afe2f2fe78ec4a0bd449b79: Status 404 returned error can't find the container with id b49229f7455d7c132291c423b45ef188ced27cdd5afe2f2fe78ec4a0bd449b79 Nov 03 02:25:00.199904 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:00.199870 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9"] Nov 03 02:25:00.218831 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:00.218804 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw"] Nov 03 02:25:00.222928 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:00.222905 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw" Nov 03 02:25:00.302601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:00.302575 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw"] Nov 03 02:25:00.305619 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:00.305587 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnlq7\" (UniqueName: \"kubernetes.io/projected/9c92a303-5b7f-438e-8839-6ae9148d340a-kube-api-access-bnlq7\") pod \"payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw\" (UID: \"9c92a303-5b7f-438e-8839-6ae9148d340a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw" Nov 03 02:25:00.306033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:00.305761 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c92a303-5b7f-438e-8839-6ae9148d340a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw\" (UID: \"9c92a303-5b7f-438e-8839-6ae9148d340a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw" Nov 03 02:25:00.309034 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:00.309003 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 02:25:00.309159 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:00.309059 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" podUID="e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 02:25:00.407705 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:00.407669 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnlq7\" (UniqueName: \"kubernetes.io/projected/9c92a303-5b7f-438e-8839-6ae9148d340a-kube-api-access-bnlq7\") pod \"payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw\" (UID: \"9c92a303-5b7f-438e-8839-6ae9148d340a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw" Nov 03 02:25:00.407891 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:00.407797 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c92a303-5b7f-438e-8839-6ae9148d340a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw\" (UID: \"9c92a303-5b7f-438e-8839-6ae9148d340a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw" Nov 03 02:25:00.408778 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:00.408739 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c92a303-5b7f-438e-8839-6ae9148d340a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw\" (UID: \"9c92a303-5b7f-438e-8839-6ae9148d340a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw" Nov 03 02:25:00.472311 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:00.471104 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnlq7\" (UniqueName: \"kubernetes.io/projected/9c92a303-5b7f-438e-8839-6ae9148d340a-kube-api-access-bnlq7\") pod \"payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw\" (UID: \"9c92a303-5b7f-438e-8839-6ae9148d340a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw" Nov 03 02:25:00.504988 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:00.504950 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9" event={"ID":"c97f240a-07b4-4e32-be49-4b7a9f077012","Type":"ContainerStarted","Data":"32ad3e16c2a4c349a6c30cae5d920bd2678305a20ce5a4a956a4964777188c73"} Nov 03 02:25:00.505177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:00.505009 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9" event={"ID":"c97f240a-07b4-4e32-be49-4b7a9f077012","Type":"ContainerStarted","Data":"b49229f7455d7c132291c423b45ef188ced27cdd5afe2f2fe78ec4a0bd449b79"} Nov 03 02:25:00.505177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:00.505051 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9" Nov 03 02:25:00.541796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:00.541770 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:25:00.581329 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:00.581296 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw" Nov 03 02:25:00.965216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:00.961929 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9" podStartSLOduration=1.961910278 podStartE2EDuration="1.961910278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:25:00.62696226 +0000 UTC m=+1982.311071416" watchObservedRunningTime="2026-11-03 02:25:00.961910278 +0000 UTC m=+1982.646019434" Nov 03 02:25:00.971789 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:00.971642 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw"] Nov 03 02:25:00.984866 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:25:00.984806 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c92a303_5b7f_438e_8839_6ae9148d340a.slice/crio-fe67ce2b15b00eab8d924d5f2843e9f91592b0208e6b4efa9a7c0717c9d3f3ff WatchSource:0}: Error finding container fe67ce2b15b00eab8d924d5f2843e9f91592b0208e6b4efa9a7c0717c9d3f3ff: Status 404 returned error can't find the container with id fe67ce2b15b00eab8d924d5f2843e9f91592b0208e6b4efa9a7c0717c9d3f3ff Nov 03 02:25:01.179631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:01.179596 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:01.305093 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:01.304996 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 02:25:01.305093 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:01.305058 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" podUID="e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 02:25:01.514921 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:01.514885 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw" event={"ID":"9c92a303-5b7f-438e-8839-6ae9148d340a","Type":"ContainerStarted","Data":"4068599889c8f29b131d5b7da710cb3af3d5409ea8f8ac4bae6cfa3779677a2d"} Nov 03 02:25:01.515426 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:01.514942 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw" event={"ID":"9c92a303-5b7f-438e-8839-6ae9148d340a","Type":"ContainerStarted","Data":"fe67ce2b15b00eab8d924d5f2843e9f91592b0208e6b4efa9a7c0717c9d3f3ff"} Nov 03 02:25:01.515426 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:01.514971 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw" Nov 03 02:25:01.634254 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:01.634136 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw" podStartSLOduration=1.634120327 podStartE2EDuration="1.634120327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:25:01.631073171 +0000 UTC m=+1983.315182330" watchObservedRunningTime="2026-11-03 02:25:01.634120327 +0000 UTC m=+1983.318229487" Nov 03 02:25:02.304568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:02.304530 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 02:25:02.304766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:02.304591 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" podUID="e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 02:25:02.598966 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:02.598858 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn"] Nov 03 02:25:02.715657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:02.715626 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw"] Nov 03 02:25:03.091420 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:03.091389 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:03.261154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:03.261116 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:03.309225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:03.309185 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:25:03.310273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:03.310249 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:25:03.406020 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:03.405976 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:05.697819 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:05.697786 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:06.120528 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.119275 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48"] Nov 03 02:25:06.120528 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.119569 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48" podUID="8c13ed61-b2e6-40af-9e66-b9538243eaf7" containerName="payload-processing" containerID="cri-o://d3851d400aa558020288b9a13d03b8d290ff019e258cbbdd2014ba06ceb4cf43" gracePeriod=30 Nov 03 02:25:06.170484 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.170453 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9"] Nov 03 02:25:06.170663 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.170644 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9" podUID="2d7ebfc5-bca8-4c89-8dfc-fc486030f56f" containerName="payload-pre-processing" containerID="cri-o://f19e588bd0df73365707efddc1282b7e20e56f733c269f820fbab328bb9cf7fa" gracePeriod=30 Nov 03 02:25:06.284386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.284351 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:06.429233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.426799 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:06.452944 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.452918 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48" Nov 03 02:25:06.486081 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.486058 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c13ed61-b2e6-40af-9e66-b9538243eaf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c13ed61-b2e6-40af-9e66-b9538243eaf7-plugins-config-volume\") pod \"8c13ed61-b2e6-40af-9e66-b9538243eaf7\" (UID: \"8c13ed61-b2e6-40af-9e66-b9538243eaf7\") " Nov 03 02:25:06.486236 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.486206 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c13ed61-b2e6-40af-9e66-b9538243eaf7-kube-api-access-kvvvb\" (UniqueName: \"kubernetes.io/projected/8c13ed61-b2e6-40af-9e66-b9538243eaf7-kube-api-access-kvvvb\") pod \"8c13ed61-b2e6-40af-9e66-b9538243eaf7\" (UID: \"8c13ed61-b2e6-40af-9e66-b9538243eaf7\") " Nov 03 02:25:06.486513 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.486492 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c13ed61-b2e6-40af-9e66-b9538243eaf7-plugins-config-volume" pod "8c13ed61-b2e6-40af-9e66-b9538243eaf7" (UID: "8c13ed61-b2e6-40af-9e66-b9538243eaf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:25:06.486633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.486619 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c13ed61-b2e6-40af-9e66-b9538243eaf7-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:06.488142 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.488124 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c13ed61-b2e6-40af-9e66-b9538243eaf7-kube-api-access-kvvvb" pod "8c13ed61-b2e6-40af-9e66-b9538243eaf7" (UID: "8c13ed61-b2e6-40af-9e66-b9538243eaf7"). InnerVolumeSpecName "kube-api-access-kvvvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:06.491158 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.491129 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9" Nov 03 02:25:06.554633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.554604 5463 generic.go:356] "Generic (PLEG): container finished" podID="2d7ebfc5-bca8-4c89-8dfc-fc486030f56f" containerID="f19e588bd0df73365707efddc1282b7e20e56f733c269f820fbab328bb9cf7fa" exitCode=0 Nov 03 02:25:06.554746 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.554654 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9" Nov 03 02:25:06.554746 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.554695 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9" event={"ID":"2d7ebfc5-bca8-4c89-8dfc-fc486030f56f","Type":"ContainerDied","Data":"f19e588bd0df73365707efddc1282b7e20e56f733c269f820fbab328bb9cf7fa"} Nov 03 02:25:06.554846 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.554751 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9" event={"ID":"2d7ebfc5-bca8-4c89-8dfc-fc486030f56f","Type":"ContainerDied","Data":"896b627a63d9d9994245738b983da82163ae84f69c1ca3fe72e603d80f07edfc"} Nov 03 02:25:06.554846 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.554775 5463 scope.go:122] "RemoveContainer" containerID="f19e588bd0df73365707efddc1282b7e20e56f733c269f820fbab328bb9cf7fa" Nov 03 02:25:06.556483 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.556459 5463 generic.go:356] "Generic (PLEG): container finished" podID="8c13ed61-b2e6-40af-9e66-b9538243eaf7" containerID="d3851d400aa558020288b9a13d03b8d290ff019e258cbbdd2014ba06ceb4cf43" exitCode=0 Nov 03 02:25:06.556600 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.556499 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48" Nov 03 02:25:06.556600 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.556530 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48" event={"ID":"8c13ed61-b2e6-40af-9e66-b9538243eaf7","Type":"ContainerDied","Data":"d3851d400aa558020288b9a13d03b8d290ff019e258cbbdd2014ba06ceb4cf43"} Nov 03 02:25:06.556600 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.556571 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48" event={"ID":"8c13ed61-b2e6-40af-9e66-b9538243eaf7","Type":"ContainerDied","Data":"7de9fcb50d6e3a5d3b3ac2f98c28601e62b7ed9f74968f7b5b0c731ed0d87ac9"} Nov 03 02:25:06.567858 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.567839 5463 scope.go:122] "RemoveContainer" containerID="f19e588bd0df73365707efddc1282b7e20e56f733c269f820fbab328bb9cf7fa" Nov 03 02:25:06.568179 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:25:06.568151 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19e588bd0df73365707efddc1282b7e20e56f733c269f820fbab328bb9cf7fa\": container with ID starting with f19e588bd0df73365707efddc1282b7e20e56f733c269f820fbab328bb9cf7fa not found: ID does not exist" containerID="f19e588bd0df73365707efddc1282b7e20e56f733c269f820fbab328bb9cf7fa" Nov 03 02:25:06.568316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.568185 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19e588bd0df73365707efddc1282b7e20e56f733c269f820fbab328bb9cf7fa"} err="failed to get container status \"f19e588bd0df73365707efddc1282b7e20e56f733c269f820fbab328bb9cf7fa\": rpc error: code = NotFound desc = could not find container \"f19e588bd0df73365707efddc1282b7e20e56f733c269f820fbab328bb9cf7fa\": container with ID starting with f19e588bd0df73365707efddc1282b7e20e56f733c269f820fbab328bb9cf7fa not found: ID does not exist" Nov 03 02:25:06.568316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.568243 5463 scope.go:122] "RemoveContainer" containerID="d3851d400aa558020288b9a13d03b8d290ff019e258cbbdd2014ba06ceb4cf43" Nov 03 02:25:06.580287 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.580267 5463 scope.go:122] "RemoveContainer" containerID="d3851d400aa558020288b9a13d03b8d290ff019e258cbbdd2014ba06ceb4cf43" Nov 03 02:25:06.580636 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:25:06.580607 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3851d400aa558020288b9a13d03b8d290ff019e258cbbdd2014ba06ceb4cf43\": container with ID starting with d3851d400aa558020288b9a13d03b8d290ff019e258cbbdd2014ba06ceb4cf43 not found: ID does not exist" containerID="d3851d400aa558020288b9a13d03b8d290ff019e258cbbdd2014ba06ceb4cf43" Nov 03 02:25:06.580731 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.580646 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3851d400aa558020288b9a13d03b8d290ff019e258cbbdd2014ba06ceb4cf43"} err="failed to get container status \"d3851d400aa558020288b9a13d03b8d290ff019e258cbbdd2014ba06ceb4cf43\": rpc error: code = NotFound desc = could not find container \"d3851d400aa558020288b9a13d03b8d290ff019e258cbbdd2014ba06ceb4cf43\": container with ID starting with d3851d400aa558020288b9a13d03b8d290ff019e258cbbdd2014ba06ceb4cf43 not found: ID does not exist" Nov 03 02:25:06.587331 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.587309 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d7ebfc5-bca8-4c89-8dfc-fc486030f56f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7ebfc5-bca8-4c89-8dfc-fc486030f56f-plugins-config-volume\") pod \"2d7ebfc5-bca8-4c89-8dfc-fc486030f56f\" (UID: \"2d7ebfc5-bca8-4c89-8dfc-fc486030f56f\") " Nov 03 02:25:06.587433 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.587401 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7ebfc5-bca8-4c89-8dfc-fc486030f56f-kube-api-access-d95cf\" (UniqueName: \"kubernetes.io/projected/2d7ebfc5-bca8-4c89-8dfc-fc486030f56f-kube-api-access-d95cf\") pod \"2d7ebfc5-bca8-4c89-8dfc-fc486030f56f\" (UID: \"2d7ebfc5-bca8-4c89-8dfc-fc486030f56f\") " Nov 03 02:25:06.587692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.587667 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7ebfc5-bca8-4c89-8dfc-fc486030f56f-plugins-config-volume" pod "2d7ebfc5-bca8-4c89-8dfc-fc486030f56f" (UID: "2d7ebfc5-bca8-4c89-8dfc-fc486030f56f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:25:06.587850 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.587835 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7ebfc5-bca8-4c89-8dfc-fc486030f56f-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:06.587913 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.587852 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvvvb\" (UniqueName: \"kubernetes.io/projected/8c13ed61-b2e6-40af-9e66-b9538243eaf7-kube-api-access-kvvvb\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:06.589524 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.589493 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7ebfc5-bca8-4c89-8dfc-fc486030f56f-kube-api-access-d95cf" pod "2d7ebfc5-bca8-4c89-8dfc-fc486030f56f" (UID: "2d7ebfc5-bca8-4c89-8dfc-fc486030f56f"). InnerVolumeSpecName "kube-api-access-d95cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:06.695323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.690245 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d95cf\" (UniqueName: \"kubernetes.io/projected/2d7ebfc5-bca8-4c89-8dfc-fc486030f56f-kube-api-access-d95cf\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:06.695323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.694247 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48"] Nov 03 02:25:06.699252 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.699184 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f66575-db8f5884-ztj48"] Nov 03 02:25:06.976931 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.976857 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9"] Nov 03 02:25:06.979861 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:06.979837 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f66575-677489ff5f-qpql9"] Nov 03 02:25:07.541844 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:07.541809 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:07.599357 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:07.599319 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn" podUID="69e7c13a-9588-4e93-b247-1830f7ba06b7" containerName="payload-processing" containerID="cri-o://469c94a12bfc770ec9e130f8a708d2f7576418e61ead9e4780b8251f66d5bf54" gracePeriod=25 Nov 03 02:25:07.611021 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:25:07.610992 5463 logging.go:55] [core] [Channel #5233 SubChannel #5235]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.157:50052", ServerName: "10.133.0.157:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.157:50052: connect: connection refused" Nov 03 02:25:07.704540 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:07.704501 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:07.716102 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:07.716075 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw" podUID="6c39cbfb-d4b9-49c9-96be-e93ae28e9d62" containerName="payload-pre-processing" containerID="cri-o://e22725dea4a2d24c482f423d813f090f9e295d1f34044ff2131def4bdc126465" gracePeriod=25 Nov 03 02:25:07.844851 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:07.844632 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:07.891835 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:07.891811 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn" Nov 03 02:25:07.908080 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:07.908054 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69e7c13a-9588-4e93-b247-1830f7ba06b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e7c13a-9588-4e93-b247-1830f7ba06b7-plugins-config-volume\") pod \"69e7c13a-9588-4e93-b247-1830f7ba06b7\" (UID: \"69e7c13a-9588-4e93-b247-1830f7ba06b7\") " Nov 03 02:25:07.908253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:07.908116 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e7c13a-9588-4e93-b247-1830f7ba06b7-kube-api-access-kdwp7\" (UniqueName: \"kubernetes.io/projected/69e7c13a-9588-4e93-b247-1830f7ba06b7-kube-api-access-kdwp7\") pod \"69e7c13a-9588-4e93-b247-1830f7ba06b7\" (UID: \"69e7c13a-9588-4e93-b247-1830f7ba06b7\") " Nov 03 02:25:07.908494 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:07.908469 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e7c13a-9588-4e93-b247-1830f7ba06b7-plugins-config-volume" pod "69e7c13a-9588-4e93-b247-1830f7ba06b7" (UID: "69e7c13a-9588-4e93-b247-1830f7ba06b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:25:07.908724 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:07.908700 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e7c13a-9588-4e93-b247-1830f7ba06b7-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:07.910297 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:07.910276 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e7c13a-9588-4e93-b247-1830f7ba06b7-kube-api-access-kdwp7" pod "69e7c13a-9588-4e93-b247-1830f7ba06b7" (UID: "69e7c13a-9588-4e93-b247-1830f7ba06b7"). InnerVolumeSpecName "kube-api-access-kdwp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:07.988271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:07.988247 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw" Nov 03 02:25:08.010180 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.009429 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c39cbfb-d4b9-49c9-96be-e93ae28e9d62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c39cbfb-d4b9-49c9-96be-e93ae28e9d62-plugins-config-volume\") pod \"6c39cbfb-d4b9-49c9-96be-e93ae28e9d62\" (UID: \"6c39cbfb-d4b9-49c9-96be-e93ae28e9d62\") " Nov 03 02:25:08.010180 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.009484 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c39cbfb-d4b9-49c9-96be-e93ae28e9d62-kube-api-access-4ph6f\" (UniqueName: \"kubernetes.io/projected/6c39cbfb-d4b9-49c9-96be-e93ae28e9d62-kube-api-access-4ph6f\") pod \"6c39cbfb-d4b9-49c9-96be-e93ae28e9d62\" (UID: \"6c39cbfb-d4b9-49c9-96be-e93ae28e9d62\") " Nov 03 02:25:08.010180 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.009815 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c39cbfb-d4b9-49c9-96be-e93ae28e9d62-plugins-config-volume" pod "6c39cbfb-d4b9-49c9-96be-e93ae28e9d62" (UID: "6c39cbfb-d4b9-49c9-96be-e93ae28e9d62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:25:08.010180 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.010141 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c39cbfb-d4b9-49c9-96be-e93ae28e9d62-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:08.010180 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.010168 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdwp7\" (UniqueName: \"kubernetes.io/projected/69e7c13a-9588-4e93-b247-1830f7ba06b7-kube-api-access-kdwp7\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:08.011698 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.011667 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c39cbfb-d4b9-49c9-96be-e93ae28e9d62-kube-api-access-4ph6f" pod "6c39cbfb-d4b9-49c9-96be-e93ae28e9d62" (UID: "6c39cbfb-d4b9-49c9-96be-e93ae28e9d62"). InnerVolumeSpecName "kube-api-access-4ph6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:08.111523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.111437 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ph6f\" (UniqueName: \"kubernetes.io/projected/6c39cbfb-d4b9-49c9-96be-e93ae28e9d62-kube-api-access-4ph6f\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:08.173101 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.173051 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:08.326667 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.326635 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:08.443467 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.443432 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d7ebfc5-bca8-4c89-8dfc-fc486030f56f" path="/var/lib/kubelet/pods/2d7ebfc5-bca8-4c89-8dfc-fc486030f56f/volumes" Nov 03 02:25:08.443885 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.443870 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c13ed61-b2e6-40af-9e66-b9538243eaf7" path="/var/lib/kubelet/pods/8c13ed61-b2e6-40af-9e66-b9538243eaf7/volumes" Nov 03 02:25:08.573255 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.573227 5463 generic.go:356] "Generic (PLEG): container finished" podID="69e7c13a-9588-4e93-b247-1830f7ba06b7" containerID="469c94a12bfc770ec9e130f8a708d2f7576418e61ead9e4780b8251f66d5bf54" exitCode=0 Nov 03 02:25:08.573417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.573308 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn" Nov 03 02:25:08.573417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.573350 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn" event={"ID":"69e7c13a-9588-4e93-b247-1830f7ba06b7","Type":"ContainerDied","Data":"469c94a12bfc770ec9e130f8a708d2f7576418e61ead9e4780b8251f66d5bf54"} Nov 03 02:25:08.573417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.573409 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn" event={"ID":"69e7c13a-9588-4e93-b247-1830f7ba06b7","Type":"ContainerDied","Data":"0a3eb1fbcaa68d37620fe0c521f0501852399d2e806bbca47d68372be737490b"} Nov 03 02:25:08.573550 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.573440 5463 scope.go:122] "RemoveContainer" containerID="469c94a12bfc770ec9e130f8a708d2f7576418e61ead9e4780b8251f66d5bf54" Nov 03 02:25:08.575037 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.575012 5463 generic.go:356] "Generic (PLEG): container finished" podID="6c39cbfb-d4b9-49c9-96be-e93ae28e9d62" containerID="e22725dea4a2d24c482f423d813f090f9e295d1f34044ff2131def4bdc126465" exitCode=0 Nov 03 02:25:08.575162 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.575058 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw" Nov 03 02:25:08.575162 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.575063 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw" event={"ID":"6c39cbfb-d4b9-49c9-96be-e93ae28e9d62","Type":"ContainerDied","Data":"e22725dea4a2d24c482f423d813f090f9e295d1f34044ff2131def4bdc126465"} Nov 03 02:25:08.575303 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.575229 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw" event={"ID":"6c39cbfb-d4b9-49c9-96be-e93ae28e9d62","Type":"ContainerDied","Data":"26288d24c3fd7e7b940dd3c433ca63f5bdd7cded7fd4e469a53ccf65aa29b235"} Nov 03 02:25:08.588284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.588265 5463 scope.go:122] "RemoveContainer" containerID="469c94a12bfc770ec9e130f8a708d2f7576418e61ead9e4780b8251f66d5bf54" Nov 03 02:25:08.588592 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:25:08.588565 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"469c94a12bfc770ec9e130f8a708d2f7576418e61ead9e4780b8251f66d5bf54\": container with ID starting with 469c94a12bfc770ec9e130f8a708d2f7576418e61ead9e4780b8251f66d5bf54 not found: ID does not exist" containerID="469c94a12bfc770ec9e130f8a708d2f7576418e61ead9e4780b8251f66d5bf54" Nov 03 02:25:08.588665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.588602 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"469c94a12bfc770ec9e130f8a708d2f7576418e61ead9e4780b8251f66d5bf54"} err="failed to get container status \"469c94a12bfc770ec9e130f8a708d2f7576418e61ead9e4780b8251f66d5bf54\": rpc error: code = NotFound desc = could not find container \"469c94a12bfc770ec9e130f8a708d2f7576418e61ead9e4780b8251f66d5bf54\": container with ID starting with 469c94a12bfc770ec9e130f8a708d2f7576418e61ead9e4780b8251f66d5bf54 not found: ID does not exist" Nov 03 02:25:08.588665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.588626 5463 scope.go:122] "RemoveContainer" containerID="e22725dea4a2d24c482f423d813f090f9e295d1f34044ff2131def4bdc126465" Nov 03 02:25:08.598309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.598288 5463 scope.go:122] "RemoveContainer" containerID="e22725dea4a2d24c482f423d813f090f9e295d1f34044ff2131def4bdc126465" Nov 03 02:25:08.598597 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:25:08.598578 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e22725dea4a2d24c482f423d813f090f9e295d1f34044ff2131def4bdc126465\": container with ID starting with e22725dea4a2d24c482f423d813f090f9e295d1f34044ff2131def4bdc126465 not found: ID does not exist" containerID="e22725dea4a2d24c482f423d813f090f9e295d1f34044ff2131def4bdc126465" Nov 03 02:25:08.598661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.598602 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e22725dea4a2d24c482f423d813f090f9e295d1f34044ff2131def4bdc126465"} err="failed to get container status \"e22725dea4a2d24c482f423d813f090f9e295d1f34044ff2131def4bdc126465\": rpc error: code = NotFound desc = could not find container \"e22725dea4a2d24c482f423d813f090f9e295d1f34044ff2131def4bdc126465\": container with ID starting with e22725dea4a2d24c482f423d813f090f9e295d1f34044ff2131def4bdc126465 not found: ID does not exist" Nov 03 02:25:08.610995 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.610973 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn" podUID="69e7c13a-9588-4e93-b247-1830f7ba06b7" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.157:50052\" within 1s: context deadline exceeded" Nov 03 02:25:08.739555 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.739480 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw"] Nov 03 02:25:08.743360 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.743337 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3bbfc4-7f9696f695-dkmcw"] Nov 03 02:25:08.824492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.824452 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:08.852577 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.852546 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn"] Nov 03 02:25:08.855556 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:08.855535 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3bbfc4-85c54f45f-lv2fn"] Nov 03 02:25:09.688858 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:09.688812 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:10.185237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:10.185182 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f"] Nov 03 02:25:10.185644 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:10.185435 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" podUID="a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" containerName="istio-proxy" containerID="cri-o://1581ecb527924e3d11fd74ba28f7b3ebced6afefd88445db0a33541ae86abadd" gracePeriod=30 Nov 03 02:25:10.195526 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:10.195491 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:10.448485 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:10.448388 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69e7c13a-9588-4e93-b247-1830f7ba06b7" path="/var/lib/kubelet/pods/69e7c13a-9588-4e93-b247-1830f7ba06b7/volumes" Nov 03 02:25:10.449310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:10.449288 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c39cbfb-d4b9-49c9-96be-e93ae28e9d62" path="/var/lib/kubelet/pods/6c39cbfb-d4b9-49c9-96be-e93ae28e9d62/volumes" Nov 03 02:25:11.009122 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:11.009088 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:11.518177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:11.518147 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9" Nov 03 02:25:11.989947 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:11.989913 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4"] Nov 03 02:25:11.990174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:11.990151 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" podUID="7130486e-0af7-40d3-8279-695ee3927707" containerName="maas-api" containerID="cri-o://fec4725423dec6f1b0d186e1511dad72aaea4db2b42e19a824d93ed371bfc47e" gracePeriod=30 Nov 03 02:25:12.111343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.111311 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:12.160019 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.159978 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" podUID="7130486e-0af7-40d3-8279-695ee3927707" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.151:8443/health\": dial tcp 10.133.0.151:8443: connect: connection refused" Nov 03 02:25:12.283759 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.283727 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" Nov 03 02:25:12.370002 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.369960 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7130486e-0af7-40d3-8279-695ee3927707-kube-api-access-s4xxx\" (UniqueName: \"kubernetes.io/projected/7130486e-0af7-40d3-8279-695ee3927707-kube-api-access-s4xxx\") pod \"7130486e-0af7-40d3-8279-695ee3927707\" (UID: \"7130486e-0af7-40d3-8279-695ee3927707\") " Nov 03 02:25:12.370210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.370029 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7130486e-0af7-40d3-8279-695ee3927707-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7130486e-0af7-40d3-8279-695ee3927707-metrics-certs\") pod \"7130486e-0af7-40d3-8279-695ee3927707\" (UID: \"7130486e-0af7-40d3-8279-695ee3927707\") " Nov 03 02:25:12.370210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.370157 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7130486e-0af7-40d3-8279-695ee3927707-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7130486e-0af7-40d3-8279-695ee3927707-openshift-service-ca\") pod \"7130486e-0af7-40d3-8279-695ee3927707\" (UID: \"7130486e-0af7-40d3-8279-695ee3927707\") " Nov 03 02:25:12.370362 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.370228 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7130486e-0af7-40d3-8279-695ee3927707-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7130486e-0af7-40d3-8279-695ee3927707-maas-api-tls\") pod \"7130486e-0af7-40d3-8279-695ee3927707\" (UID: \"7130486e-0af7-40d3-8279-695ee3927707\") " Nov 03 02:25:12.370591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.370564 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7130486e-0af7-40d3-8279-695ee3927707-openshift-service-ca" pod "7130486e-0af7-40d3-8279-695ee3927707" (UID: "7130486e-0af7-40d3-8279-695ee3927707"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:25:12.370945 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.370890 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7130486e-0af7-40d3-8279-695ee3927707-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:12.372375 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.372345 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7130486e-0af7-40d3-8279-695ee3927707-metrics-certs" pod "7130486e-0af7-40d3-8279-695ee3927707" (UID: "7130486e-0af7-40d3-8279-695ee3927707"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:25:12.372467 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.372409 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7130486e-0af7-40d3-8279-695ee3927707-maas-api-tls" pod "7130486e-0af7-40d3-8279-695ee3927707" (UID: "7130486e-0af7-40d3-8279-695ee3927707"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:25:12.372576 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.372558 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7130486e-0af7-40d3-8279-695ee3927707-kube-api-access-s4xxx" pod "7130486e-0af7-40d3-8279-695ee3927707" (UID: "7130486e-0af7-40d3-8279-695ee3927707"). InnerVolumeSpecName "kube-api-access-s4xxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:12.472060 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.472025 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7130486e-0af7-40d3-8279-695ee3927707-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:12.472060 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.472060 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4xxx\" (UniqueName: \"kubernetes.io/projected/7130486e-0af7-40d3-8279-695ee3927707-kube-api-access-s4xxx\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:12.472306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.472075 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7130486e-0af7-40d3-8279-695ee3927707-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:12.526400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.526327 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw" Nov 03 02:25:12.540451 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.540410 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6"] Nov 03 02:25:12.540664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.540631 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6" podUID="7fdb6b65-5f4e-4058-b392-0539939099fb" containerName="payload-processing" containerID="cri-o://dbbd9d9a31746b75f5aaa74830ee3a36e4cb6fb27ae8b85ab66c304453cd4f50" gracePeriod=30 Nov 03 02:25:12.594518 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.594491 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr"] Nov 03 02:25:12.594710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.594690 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr" podUID="a33f6aaf-7588-488b-8b31-9ba405917c08" containerName="payload-pre-processing" containerID="cri-o://d078d7a04c28d7eeef67a19d3b3dd876d51b1082b5572d0e26b6f72f15dcde6b" gracePeriod=30 Nov 03 02:25:12.606216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.606177 5463 generic.go:356] "Generic (PLEG): container finished" podID="7130486e-0af7-40d3-8279-695ee3927707" containerID="fec4725423dec6f1b0d186e1511dad72aaea4db2b42e19a824d93ed371bfc47e" exitCode=0 Nov 03 02:25:12.606341 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.606216 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" event={"ID":"7130486e-0af7-40d3-8279-695ee3927707","Type":"ContainerDied","Data":"fec4725423dec6f1b0d186e1511dad72aaea4db2b42e19a824d93ed371bfc47e"} Nov 03 02:25:12.606341 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.606245 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" Nov 03 02:25:12.606341 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.606266 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4" event={"ID":"7130486e-0af7-40d3-8279-695ee3927707","Type":"ContainerDied","Data":"8ac4599d44b7854f3b6e035603ca3655664199d5dddfdaa4fc82519fd2e43f4b"} Nov 03 02:25:12.606341 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.606288 5463 scope.go:122] "RemoveContainer" containerID="fec4725423dec6f1b0d186e1511dad72aaea4db2b42e19a824d93ed371bfc47e" Nov 03 02:25:12.643048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.643006 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr" podUID="a33f6aaf-7588-488b-8b31-9ba405917c08" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.172:9004: connect: connection refused" Nov 03 02:25:12.724445 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.718622 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:12.727000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.726980 5463 scope.go:122] "RemoveContainer" containerID="fec4725423dec6f1b0d186e1511dad72aaea4db2b42e19a824d93ed371bfc47e" Nov 03 02:25:12.727376 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:25:12.727353 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fec4725423dec6f1b0d186e1511dad72aaea4db2b42e19a824d93ed371bfc47e\": container with ID starting with fec4725423dec6f1b0d186e1511dad72aaea4db2b42e19a824d93ed371bfc47e not found: ID does not exist" containerID="fec4725423dec6f1b0d186e1511dad72aaea4db2b42e19a824d93ed371bfc47e" Nov 03 02:25:12.727469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.727384 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec4725423dec6f1b0d186e1511dad72aaea4db2b42e19a824d93ed371bfc47e"} err="failed to get container status \"fec4725423dec6f1b0d186e1511dad72aaea4db2b42e19a824d93ed371bfc47e\": rpc error: code = NotFound desc = could not find container \"fec4725423dec6f1b0d186e1511dad72aaea4db2b42e19a824d93ed371bfc47e\": container with ID starting with fec4725423dec6f1b0d186e1511dad72aaea4db2b42e19a824d93ed371bfc47e not found: ID does not exist" Nov 03 02:25:12.861851 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.861816 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:12.882623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.882555 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6" Nov 03 02:25:12.921381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.921358 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr" Nov 03 02:25:12.979843 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.979812 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fdb6b65-5f4e-4058-b392-0539939099fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fdb6b65-5f4e-4058-b392-0539939099fb-plugins-config-volume\") pod \"7fdb6b65-5f4e-4058-b392-0539939099fb\" (UID: \"7fdb6b65-5f4e-4058-b392-0539939099fb\") " Nov 03 02:25:12.980036 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.980005 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33f6aaf-7588-488b-8b31-9ba405917c08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33f6aaf-7588-488b-8b31-9ba405917c08-plugins-config-volume\") pod \"a33f6aaf-7588-488b-8b31-9ba405917c08\" (UID: \"a33f6aaf-7588-488b-8b31-9ba405917c08\") " Nov 03 02:25:12.980146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.980077 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33f6aaf-7588-488b-8b31-9ba405917c08-kube-api-access-z59dl\" (UniqueName: \"kubernetes.io/projected/a33f6aaf-7588-488b-8b31-9ba405917c08-kube-api-access-z59dl\") pod \"a33f6aaf-7588-488b-8b31-9ba405917c08\" (UID: \"a33f6aaf-7588-488b-8b31-9ba405917c08\") " Nov 03 02:25:12.980146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.980109 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fdb6b65-5f4e-4058-b392-0539939099fb-kube-api-access-pnbg6\" (UniqueName: \"kubernetes.io/projected/7fdb6b65-5f4e-4058-b392-0539939099fb-kube-api-access-pnbg6\") pod \"7fdb6b65-5f4e-4058-b392-0539939099fb\" (UID: \"7fdb6b65-5f4e-4058-b392-0539939099fb\") " Nov 03 02:25:12.980661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.980637 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fdb6b65-5f4e-4058-b392-0539939099fb-plugins-config-volume" pod "7fdb6b65-5f4e-4058-b392-0539939099fb" (UID: "7fdb6b65-5f4e-4058-b392-0539939099fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:25:12.980776 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.980622 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33f6aaf-7588-488b-8b31-9ba405917c08-plugins-config-volume" pod "a33f6aaf-7588-488b-8b31-9ba405917c08" (UID: "a33f6aaf-7588-488b-8b31-9ba405917c08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:25:12.982973 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.982935 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fdb6b65-5f4e-4058-b392-0539939099fb-kube-api-access-pnbg6" pod "7fdb6b65-5f4e-4058-b392-0539939099fb" (UID: "7fdb6b65-5f4e-4058-b392-0539939099fb"). InnerVolumeSpecName "kube-api-access-pnbg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:12.983583 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:12.983556 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33f6aaf-7588-488b-8b31-9ba405917c08-kube-api-access-z59dl" pod "a33f6aaf-7588-488b-8b31-9ba405917c08" (UID: "a33f6aaf-7588-488b-8b31-9ba405917c08"). InnerVolumeSpecName "kube-api-access-z59dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:13.009020 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:25:13.008987 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:25:13.109319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.096759 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z59dl\" (UniqueName: \"kubernetes.io/projected/a33f6aaf-7588-488b-8b31-9ba405917c08-kube-api-access-z59dl\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:13.109319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.096796 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnbg6\" (UniqueName: \"kubernetes.io/projected/7fdb6b65-5f4e-4058-b392-0539939099fb-kube-api-access-pnbg6\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:13.109319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.096815 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fdb6b65-5f4e-4058-b392-0539939099fb-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:13.109319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.096830 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33f6aaf-7588-488b-8b31-9ba405917c08-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:13.165752 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.165712 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4"] Nov 03 02:25:13.169247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.169217 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b3bbbc-69f6f7c8d9-kmlj4"] Nov 03 02:25:13.614419 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.614385 5463 generic.go:356] "Generic (PLEG): container finished" podID="7fdb6b65-5f4e-4058-b392-0539939099fb" containerID="dbbd9d9a31746b75f5aaa74830ee3a36e4cb6fb27ae8b85ab66c304453cd4f50" exitCode=0 Nov 03 02:25:13.614872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.614430 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6" Nov 03 02:25:13.614872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.614463 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6" event={"ID":"7fdb6b65-5f4e-4058-b392-0539939099fb","Type":"ContainerDied","Data":"dbbd9d9a31746b75f5aaa74830ee3a36e4cb6fb27ae8b85ab66c304453cd4f50"} Nov 03 02:25:13.614872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.614519 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6" event={"ID":"7fdb6b65-5f4e-4058-b392-0539939099fb","Type":"ContainerDied","Data":"a8c110ade07c0d2a2aef487849de0778f66f70c7a2474dc2ea7cb6defb6b0223"} Nov 03 02:25:13.614872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.614546 5463 scope.go:122] "RemoveContainer" containerID="dbbd9d9a31746b75f5aaa74830ee3a36e4cb6fb27ae8b85ab66c304453cd4f50" Nov 03 02:25:13.617079 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.617055 5463 generic.go:356] "Generic (PLEG): container finished" podID="a33f6aaf-7588-488b-8b31-9ba405917c08" containerID="d078d7a04c28d7eeef67a19d3b3dd876d51b1082b5572d0e26b6f72f15dcde6b" exitCode=0 Nov 03 02:25:13.617180 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.617111 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr" Nov 03 02:25:13.617180 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.617130 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr" event={"ID":"a33f6aaf-7588-488b-8b31-9ba405917c08","Type":"ContainerDied","Data":"d078d7a04c28d7eeef67a19d3b3dd876d51b1082b5572d0e26b6f72f15dcde6b"} Nov 03 02:25:13.617304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.617171 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr" event={"ID":"a33f6aaf-7588-488b-8b31-9ba405917c08","Type":"ContainerDied","Data":"b6fb6e247bc066316bdc2bbd363b6b990f4b65993d44e6041908a78965994d21"} Nov 03 02:25:13.626687 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.626666 5463 scope.go:122] "RemoveContainer" containerID="dbbd9d9a31746b75f5aaa74830ee3a36e4cb6fb27ae8b85ab66c304453cd4f50" Nov 03 02:25:13.627002 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:25:13.626982 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbbd9d9a31746b75f5aaa74830ee3a36e4cb6fb27ae8b85ab66c304453cd4f50\": container with ID starting with dbbd9d9a31746b75f5aaa74830ee3a36e4cb6fb27ae8b85ab66c304453cd4f50 not found: ID does not exist" containerID="dbbd9d9a31746b75f5aaa74830ee3a36e4cb6fb27ae8b85ab66c304453cd4f50" Nov 03 02:25:13.627078 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.627010 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbd9d9a31746b75f5aaa74830ee3a36e4cb6fb27ae8b85ab66c304453cd4f50"} err="failed to get container status \"dbbd9d9a31746b75f5aaa74830ee3a36e4cb6fb27ae8b85ab66c304453cd4f50\": rpc error: code = NotFound desc = could not find container \"dbbd9d9a31746b75f5aaa74830ee3a36e4cb6fb27ae8b85ab66c304453cd4f50\": container with ID starting with dbbd9d9a31746b75f5aaa74830ee3a36e4cb6fb27ae8b85ab66c304453cd4f50 not found: ID does not exist" Nov 03 02:25:13.627078 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.627028 5463 scope.go:122] "RemoveContainer" containerID="d078d7a04c28d7eeef67a19d3b3dd876d51b1082b5572d0e26b6f72f15dcde6b" Nov 03 02:25:13.636090 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.636075 5463 scope.go:122] "RemoveContainer" containerID="d078d7a04c28d7eeef67a19d3b3dd876d51b1082b5572d0e26b6f72f15dcde6b" Nov 03 02:25:13.636403 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:25:13.636383 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d078d7a04c28d7eeef67a19d3b3dd876d51b1082b5572d0e26b6f72f15dcde6b\": container with ID starting with d078d7a04c28d7eeef67a19d3b3dd876d51b1082b5572d0e26b6f72f15dcde6b not found: ID does not exist" containerID="d078d7a04c28d7eeef67a19d3b3dd876d51b1082b5572d0e26b6f72f15dcde6b" Nov 03 02:25:13.636467 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.636412 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d078d7a04c28d7eeef67a19d3b3dd876d51b1082b5572d0e26b6f72f15dcde6b"} err="failed to get container status \"d078d7a04c28d7eeef67a19d3b3dd876d51b1082b5572d0e26b6f72f15dcde6b\": rpc error: code = NotFound desc = could not find container \"d078d7a04c28d7eeef67a19d3b3dd876d51b1082b5572d0e26b6f72f15dcde6b\": container with ID starting with d078d7a04c28d7eeef67a19d3b3dd876d51b1082b5572d0e26b6f72f15dcde6b not found: ID does not exist" Nov 03 02:25:13.780480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.780444 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr"] Nov 03 02:25:13.784574 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.784548 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b3bbbc-87df94fcf-bwffr"] Nov 03 02:25:13.881289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.881176 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:25:13.881289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.881272 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" podUID="a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:25:13.896168 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.896137 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6"] Nov 03 02:25:13.900225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:13.900204 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b3bbbc-df5df9874-xs4f6"] Nov 03 02:25:14.166495 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:14.166460 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:14.326339 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:14.326301 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:14.443635 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:14.443555 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7130486e-0af7-40d3-8279-695ee3927707" path="/var/lib/kubelet/pods/7130486e-0af7-40d3-8279-695ee3927707/volumes" Nov 03 02:25:14.443973 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:14.443958 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fdb6b65-5f4e-4058-b392-0539939099fb" path="/var/lib/kubelet/pods/7fdb6b65-5f4e-4058-b392-0539939099fb/volumes" Nov 03 02:25:14.444387 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:14.444373 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a33f6aaf-7588-488b-8b31-9ba405917c08" path="/var/lib/kubelet/pods/a33f6aaf-7588-488b-8b31-9ba405917c08/volumes" Nov 03 02:25:15.463439 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.463379 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:25:15.529557 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.529527 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-kube-api-access-whbm5\" (UniqueName: \"kubernetes.io/projected/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-kube-api-access-whbm5\") pod \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " Nov 03 02:25:15.529710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.529571 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-workload-certs\") pod \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " Nov 03 02:25:15.529710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.529615 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-token\" (UniqueName: \"kubernetes.io/projected/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-token\") pod \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " Nov 03 02:25:15.529710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.529660 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istiod-ca-cert\") pod \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " Nov 03 02:25:15.529855 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.529707 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-data\") pod \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " Nov 03 02:25:15.529855 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.529776 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-credential-socket\") pod \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " Nov 03 02:25:15.529975 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.529861 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-workload-socket\") pod \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " Nov 03 02:25:15.529975 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.529953 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-envoy\") pod \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " Nov 03 02:25:15.530105 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.529946 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-workload-certs" pod "a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" (UID: "a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:15.530105 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.529998 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-podinfo\") pod \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\" (UID: \"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd\") " Nov 03 02:25:15.530105 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.530085 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-data" pod "a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" (UID: "a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:15.530269 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.530132 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istiod-ca-cert" pod "a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" (UID: "a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:25:15.530269 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.530245 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-workload-socket" pod "a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" (UID: "a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:15.530418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.530392 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-credential-socket" pod "a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" (UID: "a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:15.530842 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.530818 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:15.530968 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.530844 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:15.530968 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.530859 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:15.530968 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.530873 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:15.530968 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.530888 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:15.531944 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.531925 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-kube-api-access-whbm5" pod "a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" (UID: "a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd"). InnerVolumeSpecName "kube-api-access-whbm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:15.532154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.532118 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-envoy" pod "a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" (UID: "a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:15.532301 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.532279 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-token" pod "a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" (UID: "a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:15.532524 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.532496 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-podinfo" pod "a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" (UID: "a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:25:15.632397 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.632368 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:15.632397 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.632394 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:15.632397 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.632404 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whbm5\" (UniqueName: \"kubernetes.io/projected/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-kube-api-access-whbm5\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:15.632681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.632414 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:15.635807 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.635775 5463 generic.go:356] "Generic (PLEG): container finished" podID="a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" containerID="1581ecb527924e3d11fd74ba28f7b3ebced6afefd88445db0a33541ae86abadd" exitCode=0 Nov 03 02:25:15.635941 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.635832 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" Nov 03 02:25:15.635941 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.635854 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" event={"ID":"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd","Type":"ContainerDied","Data":"1581ecb527924e3d11fd74ba28f7b3ebced6afefd88445db0a33541ae86abadd"} Nov 03 02:25:15.635941 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.635906 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f" event={"ID":"a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd","Type":"ContainerDied","Data":"c4b4888c899512e766cbf15df08dde6f976ee659de5da11d24f57c467a3f30da"} Nov 03 02:25:15.635941 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.635931 5463 scope.go:122] "RemoveContainer" containerID="1581ecb527924e3d11fd74ba28f7b3ebced6afefd88445db0a33541ae86abadd" Nov 03 02:25:15.641647 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.641459 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:15.648367 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.648346 5463 scope.go:122] "RemoveContainer" containerID="99dca0b9f302869d45fae0b9555ae0f00430ab14f5c2a3ff1b6d01822a39155a" Nov 03 02:25:15.658179 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.658160 5463 scope.go:122] "RemoveContainer" containerID="1581ecb527924e3d11fd74ba28f7b3ebced6afefd88445db0a33541ae86abadd" Nov 03 02:25:15.658526 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:25:15.658501 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1581ecb527924e3d11fd74ba28f7b3ebced6afefd88445db0a33541ae86abadd\": container with ID starting with 1581ecb527924e3d11fd74ba28f7b3ebced6afefd88445db0a33541ae86abadd not found: ID does not exist" containerID="1581ecb527924e3d11fd74ba28f7b3ebced6afefd88445db0a33541ae86abadd" Nov 03 02:25:15.658631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.658536 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1581ecb527924e3d11fd74ba28f7b3ebced6afefd88445db0a33541ae86abadd"} err="failed to get container status \"1581ecb527924e3d11fd74ba28f7b3ebced6afefd88445db0a33541ae86abadd\": rpc error: code = NotFound desc = could not find container \"1581ecb527924e3d11fd74ba28f7b3ebced6afefd88445db0a33541ae86abadd\": container with ID starting with 1581ecb527924e3d11fd74ba28f7b3ebced6afefd88445db0a33541ae86abadd not found: ID does not exist" Nov 03 02:25:15.658631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.658561 5463 scope.go:122] "RemoveContainer" containerID="99dca0b9f302869d45fae0b9555ae0f00430ab14f5c2a3ff1b6d01822a39155a" Nov 03 02:25:15.658834 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:25:15.658814 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99dca0b9f302869d45fae0b9555ae0f00430ab14f5c2a3ff1b6d01822a39155a\": container with ID starting with 99dca0b9f302869d45fae0b9555ae0f00430ab14f5c2a3ff1b6d01822a39155a not found: ID does not exist" containerID="99dca0b9f302869d45fae0b9555ae0f00430ab14f5c2a3ff1b6d01822a39155a" Nov 03 02:25:15.658902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.658839 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99dca0b9f302869d45fae0b9555ae0f00430ab14f5c2a3ff1b6d01822a39155a"} err="failed to get container status \"99dca0b9f302869d45fae0b9555ae0f00430ab14f5c2a3ff1b6d01822a39155a\": rpc error: code = NotFound desc = could not find container \"99dca0b9f302869d45fae0b9555ae0f00430ab14f5c2a3ff1b6d01822a39155a\": container with ID starting with 99dca0b9f302869d45fae0b9555ae0f00430ab14f5c2a3ff1b6d01822a39155a not found: ID does not exist" Nov 03 02:25:15.804100 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.804014 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f"] Nov 03 02:25:15.806834 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:15.806812 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f66575-openshift-default-5dbbf777d5-2258f"] Nov 03 02:25:16.115148 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:16.115065 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84"] Nov 03 02:25:16.115352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:16.115326 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" podUID="d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" containerName="istio-proxy" containerID="cri-o://15724a1b56caa393d9eab37121e76db3f2d8bf36578a5dd60531f4d07982c30b" gracePeriod=30 Nov 03 02:25:16.175910 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:16.175832 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:16.444696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:16.444661 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" path="/var/lib/kubelet/pods/a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd/volumes" Nov 03 02:25:16.990572 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:16.990533 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:17.655886 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:17.655848 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:21.367246 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.367172 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:21.414756 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.414731 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:25:21.518289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.518261 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-envoy\") pod \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " Nov 03 02:25:21.518447 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.518322 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-credential-socket\") pod \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " Nov 03 02:25:21.518447 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.518375 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-token\" (UniqueName: \"kubernetes.io/projected/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-token\") pod \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " Nov 03 02:25:21.518531 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.518452 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istiod-ca-cert\") pod \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " Nov 03 02:25:21.518531 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.518508 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-kube-api-access-kzkpc\" (UniqueName: \"kubernetes.io/projected/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-kube-api-access-kzkpc\") pod \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " Nov 03 02:25:21.518644 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.518540 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-workload-socket\") pod \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " Nov 03 02:25:21.518644 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.518615 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-workload-certs\") pod \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " Nov 03 02:25:21.518746 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.518647 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-data\") pod \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " Nov 03 02:25:21.518746 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.518672 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-credential-socket" pod "d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" (UID: "d62ad394-77bf-4a2d-8ec1-ed0f6238b3de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:21.518746 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.518713 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-podinfo\") pod \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\" (UID: \"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de\") " Nov 03 02:25:21.518999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.518926 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istiod-ca-cert" pod "d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" (UID: "d62ad394-77bf-4a2d-8ec1-ed0f6238b3de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:25:21.518999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.518923 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-workload-certs" pod "d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" (UID: "d62ad394-77bf-4a2d-8ec1-ed0f6238b3de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:21.518999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.518955 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-workload-socket" pod "d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" (UID: "d62ad394-77bf-4a2d-8ec1-ed0f6238b3de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:21.519174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.518984 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-data" pod "d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" (UID: "d62ad394-77bf-4a2d-8ec1-ed0f6238b3de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:21.519324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.519304 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:21.519400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.519331 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:21.519400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.519346 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:21.519400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.519361 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:21.519400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.519377 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:21.520978 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.520949 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-envoy" pod "d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" (UID: "d62ad394-77bf-4a2d-8ec1-ed0f6238b3de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:21.521088 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.520988 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-podinfo" pod "d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" (UID: "d62ad394-77bf-4a2d-8ec1-ed0f6238b3de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:25:21.521088 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.520994 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-token" pod "d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" (UID: "d62ad394-77bf-4a2d-8ec1-ed0f6238b3de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:21.521088 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.521055 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-kube-api-access-kzkpc" pod "d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" (UID: "d62ad394-77bf-4a2d-8ec1-ed0f6238b3de"). InnerVolumeSpecName "kube-api-access-kzkpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:21.620622 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.620588 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:21.620622 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.620623 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzkpc\" (UniqueName: \"kubernetes.io/projected/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-kube-api-access-kzkpc\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:21.620891 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.620651 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:21.620891 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.620672 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:21.683883 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.683853 5463 generic.go:356] "Generic (PLEG): container finished" podID="d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" containerID="15724a1b56caa393d9eab37121e76db3f2d8bf36578a5dd60531f4d07982c30b" exitCode=0 Nov 03 02:25:21.684054 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.683916 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" Nov 03 02:25:21.684054 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.683941 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" event={"ID":"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de","Type":"ContainerDied","Data":"15724a1b56caa393d9eab37121e76db3f2d8bf36578a5dd60531f4d07982c30b"} Nov 03 02:25:21.684054 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.684014 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84" event={"ID":"d62ad394-77bf-4a2d-8ec1-ed0f6238b3de","Type":"ContainerDied","Data":"4e62ecc861209fa7aa927f06751066f49b99e705c7836e14b66662f8b5935254"} Nov 03 02:25:21.684054 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.684045 5463 scope.go:122] "RemoveContainer" containerID="15724a1b56caa393d9eab37121e76db3f2d8bf36578a5dd60531f4d07982c30b" Nov 03 02:25:21.695076 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.695001 5463 scope.go:122] "RemoveContainer" containerID="7b9e02876cb8942f0f9fc54b2eccdbf928cae3fd83221b37f283bb3a79523746" Nov 03 02:25:21.707405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.707387 5463 scope.go:122] "RemoveContainer" containerID="15724a1b56caa393d9eab37121e76db3f2d8bf36578a5dd60531f4d07982c30b" Nov 03 02:25:21.707682 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:25:21.707659 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15724a1b56caa393d9eab37121e76db3f2d8bf36578a5dd60531f4d07982c30b\": container with ID starting with 15724a1b56caa393d9eab37121e76db3f2d8bf36578a5dd60531f4d07982c30b not found: ID does not exist" containerID="15724a1b56caa393d9eab37121e76db3f2d8bf36578a5dd60531f4d07982c30b" Nov 03 02:25:21.707749 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.707690 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15724a1b56caa393d9eab37121e76db3f2d8bf36578a5dd60531f4d07982c30b"} err="failed to get container status \"15724a1b56caa393d9eab37121e76db3f2d8bf36578a5dd60531f4d07982c30b\": rpc error: code = NotFound desc = could not find container \"15724a1b56caa393d9eab37121e76db3f2d8bf36578a5dd60531f4d07982c30b\": container with ID starting with 15724a1b56caa393d9eab37121e76db3f2d8bf36578a5dd60531f4d07982c30b not found: ID does not exist" Nov 03 02:25:21.707749 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.707708 5463 scope.go:122] "RemoveContainer" containerID="7b9e02876cb8942f0f9fc54b2eccdbf928cae3fd83221b37f283bb3a79523746" Nov 03 02:25:21.707976 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:25:21.707960 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b9e02876cb8942f0f9fc54b2eccdbf928cae3fd83221b37f283bb3a79523746\": container with ID starting with 7b9e02876cb8942f0f9fc54b2eccdbf928cae3fd83221b37f283bb3a79523746 not found: ID does not exist" containerID="7b9e02876cb8942f0f9fc54b2eccdbf928cae3fd83221b37f283bb3a79523746" Nov 03 02:25:21.708021 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.707984 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b9e02876cb8942f0f9fc54b2eccdbf928cae3fd83221b37f283bb3a79523746"} err="failed to get container status \"7b9e02876cb8942f0f9fc54b2eccdbf928cae3fd83221b37f283bb3a79523746\": rpc error: code = NotFound desc = could not find container \"7b9e02876cb8942f0f9fc54b2eccdbf928cae3fd83221b37f283bb3a79523746\": container with ID starting with 7b9e02876cb8942f0f9fc54b2eccdbf928cae3fd83221b37f283bb3a79523746 not found: ID does not exist" Nov 03 02:25:21.852515 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.852484 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84"] Nov 03 02:25:21.857272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:21.857245 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b3bbbc-openshift-default-d68c44899-wsg84"] Nov 03 02:25:22.444871 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:22.444837 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" path="/var/lib/kubelet/pods/d62ad394-77bf-4a2d-8ec1-ed0f6238b3de/volumes" Nov 03 02:25:22.913250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:22.913185 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:25.774827 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:25.774785 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:25.928806 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:25.928771 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:26.156930 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.156885 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p"] Nov 03 02:25:26.157203 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.157162 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p" podUID="746d6093-1cc8-4ef2-90ce-22f2a1a5a973" containerName="payload-processing" containerID="cri-o://a16463f03837aaa2af0d93d636ff51035593a60377d09d8aabf350827dd30de8" gracePeriod=30 Nov 03 02:25:26.209274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.209246 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk"] Nov 03 02:25:26.209471 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.209433 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk" podUID="5303d364-11ee-429b-8545-7a498d93213b" containerName="payload-pre-processing" containerID="cri-o://0564c5ed614a5552c58d26f4ada49c5e74b2c5480fe3304d0099e61ab6dacb5d" gracePeriod=30 Nov 03 02:25:26.324327 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.324294 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:26.470272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.468429 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:26.584219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.584175 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p" Nov 03 02:25:26.589121 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.589095 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk" Nov 03 02:25:26.695382 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.695299 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/746d6093-1cc8-4ef2-90ce-22f2a1a5a973-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/746d6093-1cc8-4ef2-90ce-22f2a1a5a973-plugins-config-volume\") pod \"746d6093-1cc8-4ef2-90ce-22f2a1a5a973\" (UID: \"746d6093-1cc8-4ef2-90ce-22f2a1a5a973\") " Nov 03 02:25:26.695382 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.695337 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5303d364-11ee-429b-8545-7a498d93213b-kube-api-access-mwhpf\" (UniqueName: \"kubernetes.io/projected/5303d364-11ee-429b-8545-7a498d93213b-kube-api-access-mwhpf\") pod \"5303d364-11ee-429b-8545-7a498d93213b\" (UID: \"5303d364-11ee-429b-8545-7a498d93213b\") " Nov 03 02:25:26.695613 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.695462 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/746d6093-1cc8-4ef2-90ce-22f2a1a5a973-kube-api-access-df8dk\" (UniqueName: \"kubernetes.io/projected/746d6093-1cc8-4ef2-90ce-22f2a1a5a973-kube-api-access-df8dk\") pod \"746d6093-1cc8-4ef2-90ce-22f2a1a5a973\" (UID: \"746d6093-1cc8-4ef2-90ce-22f2a1a5a973\") " Nov 03 02:25:26.695613 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.695592 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5303d364-11ee-429b-8545-7a498d93213b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5303d364-11ee-429b-8545-7a498d93213b-plugins-config-volume\") pod \"5303d364-11ee-429b-8545-7a498d93213b\" (UID: \"5303d364-11ee-429b-8545-7a498d93213b\") " Nov 03 02:25:26.695759 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.695729 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746d6093-1cc8-4ef2-90ce-22f2a1a5a973-plugins-config-volume" pod "746d6093-1cc8-4ef2-90ce-22f2a1a5a973" (UID: "746d6093-1cc8-4ef2-90ce-22f2a1a5a973"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:25:26.695985 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.695963 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5303d364-11ee-429b-8545-7a498d93213b-plugins-config-volume" pod "5303d364-11ee-429b-8545-7a498d93213b" (UID: "5303d364-11ee-429b-8545-7a498d93213b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:25:26.696306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.696284 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5303d364-11ee-429b-8545-7a498d93213b-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:26.696306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.696306 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/746d6093-1cc8-4ef2-90ce-22f2a1a5a973-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:26.697598 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.697569 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5303d364-11ee-429b-8545-7a498d93213b-kube-api-access-mwhpf" pod "5303d364-11ee-429b-8545-7a498d93213b" (UID: "5303d364-11ee-429b-8545-7a498d93213b"). InnerVolumeSpecName "kube-api-access-mwhpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:26.697942 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.697921 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746d6093-1cc8-4ef2-90ce-22f2a1a5a973-kube-api-access-df8dk" pod "746d6093-1cc8-4ef2-90ce-22f2a1a5a973" (UID: "746d6093-1cc8-4ef2-90ce-22f2a1a5a973"). InnerVolumeSpecName "kube-api-access-df8dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:26.726577 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.726544 5463 generic.go:356] "Generic (PLEG): container finished" podID="5303d364-11ee-429b-8545-7a498d93213b" containerID="0564c5ed614a5552c58d26f4ada49c5e74b2c5480fe3304d0099e61ab6dacb5d" exitCode=0 Nov 03 02:25:26.726711 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.726636 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk" Nov 03 02:25:26.726711 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.726686 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk" event={"ID":"5303d364-11ee-429b-8545-7a498d93213b","Type":"ContainerDied","Data":"0564c5ed614a5552c58d26f4ada49c5e74b2c5480fe3304d0099e61ab6dacb5d"} Nov 03 02:25:26.726840 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.726746 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk" event={"ID":"5303d364-11ee-429b-8545-7a498d93213b","Type":"ContainerDied","Data":"0045b7f529eacb8ae1e3fcb704682e4fd3cf78177f92ea187587c2aaf0a4286e"} Nov 03 02:25:26.726840 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.726777 5463 scope.go:122] "RemoveContainer" containerID="0564c5ed614a5552c58d26f4ada49c5e74b2c5480fe3304d0099e61ab6dacb5d" Nov 03 02:25:26.728312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.728288 5463 generic.go:356] "Generic (PLEG): container finished" podID="746d6093-1cc8-4ef2-90ce-22f2a1a5a973" containerID="a16463f03837aaa2af0d93d636ff51035593a60377d09d8aabf350827dd30de8" exitCode=0 Nov 03 02:25:26.728434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.728326 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p" Nov 03 02:25:26.728434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.728359 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p" event={"ID":"746d6093-1cc8-4ef2-90ce-22f2a1a5a973","Type":"ContainerDied","Data":"a16463f03837aaa2af0d93d636ff51035593a60377d09d8aabf350827dd30de8"} Nov 03 02:25:26.728434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.728409 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p" event={"ID":"746d6093-1cc8-4ef2-90ce-22f2a1a5a973","Type":"ContainerDied","Data":"9ec1c39aaa3e95a4a477429210e286383ca8016d87a3687ba193f411e6f4c077"} Nov 03 02:25:26.738038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.738010 5463 scope.go:122] "RemoveContainer" containerID="0564c5ed614a5552c58d26f4ada49c5e74b2c5480fe3304d0099e61ab6dacb5d" Nov 03 02:25:26.738557 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:25:26.738531 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0564c5ed614a5552c58d26f4ada49c5e74b2c5480fe3304d0099e61ab6dacb5d\": container with ID starting with 0564c5ed614a5552c58d26f4ada49c5e74b2c5480fe3304d0099e61ab6dacb5d not found: ID does not exist" containerID="0564c5ed614a5552c58d26f4ada49c5e74b2c5480fe3304d0099e61ab6dacb5d" Nov 03 02:25:26.738680 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.738567 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0564c5ed614a5552c58d26f4ada49c5e74b2c5480fe3304d0099e61ab6dacb5d"} err="failed to get container status \"0564c5ed614a5552c58d26f4ada49c5e74b2c5480fe3304d0099e61ab6dacb5d\": rpc error: code = NotFound desc = could not find container \"0564c5ed614a5552c58d26f4ada49c5e74b2c5480fe3304d0099e61ab6dacb5d\": container with ID starting with 0564c5ed614a5552c58d26f4ada49c5e74b2c5480fe3304d0099e61ab6dacb5d not found: ID does not exist" Nov 03 02:25:26.738680 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.738592 5463 scope.go:122] "RemoveContainer" containerID="a16463f03837aaa2af0d93d636ff51035593a60377d09d8aabf350827dd30de8" Nov 03 02:25:26.752263 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.752244 5463 scope.go:122] "RemoveContainer" containerID="a16463f03837aaa2af0d93d636ff51035593a60377d09d8aabf350827dd30de8" Nov 03 02:25:26.752575 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:25:26.752550 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a16463f03837aaa2af0d93d636ff51035593a60377d09d8aabf350827dd30de8\": container with ID starting with a16463f03837aaa2af0d93d636ff51035593a60377d09d8aabf350827dd30de8 not found: ID does not exist" containerID="a16463f03837aaa2af0d93d636ff51035593a60377d09d8aabf350827dd30de8" Nov 03 02:25:26.752657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.752583 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16463f03837aaa2af0d93d636ff51035593a60377d09d8aabf350827dd30de8"} err="failed to get container status \"a16463f03837aaa2af0d93d636ff51035593a60377d09d8aabf350827dd30de8\": rpc error: code = NotFound desc = could not find container \"a16463f03837aaa2af0d93d636ff51035593a60377d09d8aabf350827dd30de8\": container with ID starting with a16463f03837aaa2af0d93d636ff51035593a60377d09d8aabf350827dd30de8 not found: ID does not exist" Nov 03 02:25:26.809212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.805504 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df8dk\" (UniqueName: \"kubernetes.io/projected/746d6093-1cc8-4ef2-90ce-22f2a1a5a973-kube-api-access-df8dk\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:26.809212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.805545 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwhpf\" (UniqueName: \"kubernetes.io/projected/5303d364-11ee-429b-8545-7a498d93213b-kube-api-access-mwhpf\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:26.930626 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.930587 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p"] Nov 03 02:25:26.933655 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:26.933634 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad4591-77f596b5b4-9b89p"] Nov 03 02:25:27.042424 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:27.042339 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk"] Nov 03 02:25:27.047365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:27.047336 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad4591-6979cdb98c-xqgbk"] Nov 03 02:25:27.743061 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:27.743025 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:28.386462 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:28.386432 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:28.444952 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:28.444919 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5303d364-11ee-429b-8545-7a498d93213b" path="/var/lib/kubelet/pods/5303d364-11ee-429b-8545-7a498d93213b/volumes" Nov 03 02:25:28.445337 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:28.445323 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="746d6093-1cc8-4ef2-90ce-22f2a1a5a973" path="/var/lib/kubelet/pods/746d6093-1cc8-4ef2-90ce-22f2a1a5a973/volumes" Nov 03 02:25:28.534832 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:28.534794 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:29.989854 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:29.989822 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:30.523325 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:30.523288 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:30.529155 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:30.529134 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd"] Nov 03 02:25:30.529375 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:30.529355 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" podUID="255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" containerName="istio-proxy" containerID="cri-o://fb5c65ed4de6baf0dd3d2cdf6d7f2c33abc9343b79e138f1cfb283fa31329399" gracePeriod=30 Nov 03 02:25:32.729404 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:32.729374 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:33.507981 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:33.507942 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:34.036579 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.036549 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd"] Nov 03 02:25:34.037157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037143 5463 cpu_manager.go:414] "containerMap: removing container" podUID="7fdb6b65-5f4e-4058-b392-0539939099fb" containerName="payload-processing" Nov 03 02:25:34.037234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037158 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fdb6b65-5f4e-4058-b392-0539939099fb" containerName="payload-processing" Nov 03 02:25:34.037234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037173 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" containerName="istio-proxy" Nov 03 02:25:34.037234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037179 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" containerName="istio-proxy" Nov 03 02:25:34.037234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037201 5463 cpu_manager.go:414] "containerMap: removing container" podUID="746d6093-1cc8-4ef2-90ce-22f2a1a5a973" containerName="payload-processing" Nov 03 02:25:34.037234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037206 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="746d6093-1cc8-4ef2-90ce-22f2a1a5a973" containerName="payload-processing" Nov 03 02:25:34.037234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037222 5463 cpu_manager.go:414] "containerMap: removing container" podUID="8c13ed61-b2e6-40af-9e66-b9538243eaf7" containerName="payload-processing" Nov 03 02:25:34.037234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037227 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c13ed61-b2e6-40af-9e66-b9538243eaf7" containerName="payload-processing" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037255 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2d7ebfc5-bca8-4c89-8dfc-fc486030f56f" containerName="payload-pre-processing" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037260 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7ebfc5-bca8-4c89-8dfc-fc486030f56f" containerName="payload-pre-processing" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037272 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a33f6aaf-7588-488b-8b31-9ba405917c08" containerName="payload-pre-processing" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037277 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33f6aaf-7588-488b-8b31-9ba405917c08" containerName="payload-pre-processing" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037285 5463 cpu_manager.go:414] "containerMap: removing container" podUID="6c39cbfb-d4b9-49c9-96be-e93ae28e9d62" containerName="payload-pre-processing" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037290 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c39cbfb-d4b9-49c9-96be-e93ae28e9d62" containerName="payload-pre-processing" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037297 5463 cpu_manager.go:414] "containerMap: removing container" podUID="69e7c13a-9588-4e93-b247-1830f7ba06b7" containerName="payload-processing" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037301 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e7c13a-9588-4e93-b247-1830f7ba06b7" containerName="payload-processing" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037310 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" containerName="istio-proxy" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037314 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" containerName="istio-proxy" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037321 5463 cpu_manager.go:414] "containerMap: removing container" podUID="7130486e-0af7-40d3-8279-695ee3927707" containerName="maas-api" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037325 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7130486e-0af7-40d3-8279-695ee3927707" containerName="maas-api" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037330 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" containerName="istio-proxy" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037334 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" containerName="istio-proxy" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037339 5463 cpu_manager.go:414] "containerMap: removing container" podUID="5303d364-11ee-429b-8545-7a498d93213b" containerName="payload-pre-processing" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037343 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5303d364-11ee-429b-8545-7a498d93213b" containerName="payload-pre-processing" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037351 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" containerName="istio-proxy" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037355 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" containerName="istio-proxy" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037435 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7ebfc5-bca8-4c89-8dfc-fc486030f56f" containerName="payload-pre-processing" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037443 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33f6aaf-7588-488b-8b31-9ba405917c08" containerName="payload-pre-processing" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037451 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c39cbfb-d4b9-49c9-96be-e93ae28e9d62" containerName="payload-pre-processing" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037460 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e7c13a-9588-4e93-b247-1830f7ba06b7" containerName="payload-processing" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037467 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" containerName="istio-proxy" Nov 03 02:25:34.037464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037472 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="7130486e-0af7-40d3-8279-695ee3927707" containerName="maas-api" Nov 03 02:25:34.038182 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037477 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a80c313a-aa7c-4fd4-8c2b-7c5d2ecc7dbd" containerName="istio-proxy" Nov 03 02:25:34.038182 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037484 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="5303d364-11ee-429b-8545-7a498d93213b" containerName="payload-pre-processing" Nov 03 02:25:34.038182 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037489 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" containerName="istio-proxy" Nov 03 02:25:34.038182 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037495 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fdb6b65-5f4e-4058-b392-0539939099fb" containerName="payload-processing" Nov 03 02:25:34.038182 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037501 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d62ad394-77bf-4a2d-8ec1-ed0f6238b3de" containerName="istio-proxy" Nov 03 02:25:34.038182 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037505 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="746d6093-1cc8-4ef2-90ce-22f2a1a5a973" containerName="payload-processing" Nov 03 02:25:34.038182 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.037513 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c13ed61-b2e6-40af-9e66-b9538243eaf7" containerName="payload-processing" Nov 03 02:25:34.042646 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.042628 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.150352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.150316 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd"] Nov 03 02:25:34.165245 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.164508 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:34.195653 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.195616 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-f760d3-openshift-default-dockercfg-bz7nl\"" Nov 03 02:25:34.199689 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.199656 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-credential-socket\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.199866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.199724 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-data\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.199866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.199781 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-podinfo\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.199866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.199861 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-envoy\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.200065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.199902 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-workload-socket\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.200065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.199928 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-workload-certs\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.200175 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.200006 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d029bf9f-23a3-47c7-8eca-e343baed7d99-istiod-ca-cert\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.200300 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.200210 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q74l7\" (UniqueName: \"kubernetes.io/projected/d029bf9f-23a3-47c7-8eca-e343baed7d99-kube-api-access-q74l7\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.200300 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.200270 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-token\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.301753 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.301660 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-envoy\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.301753 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.301716 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-workload-socket\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.301753 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.301738 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-workload-certs\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.302051 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.301806 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d029bf9f-23a3-47c7-8eca-e343baed7d99-istiod-ca-cert\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.302051 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.301849 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q74l7\" (UniqueName: \"kubernetes.io/projected/d029bf9f-23a3-47c7-8eca-e343baed7d99-kube-api-access-q74l7\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.302051 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.301882 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-token\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.302051 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.302001 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-credential-socket\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.302286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.302052 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-data\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.302286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.302100 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-podinfo\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.302286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.302138 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-workload-socket\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.302286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.302176 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-workload-certs\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.302509 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.302430 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-credential-socket\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.302509 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.302453 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-data\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.302664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.302609 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d029bf9f-23a3-47c7-8eca-e343baed7d99-istiod-ca-cert\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.304415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.304383 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-envoy\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.304617 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.304601 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-podinfo\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.358603 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.358571 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-token\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.358716 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.358662 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q74l7\" (UniqueName: \"kubernetes.io/projected/d029bf9f-23a3-47c7-8eca-e343baed7d99-kube-api-access-q74l7\") pod \"e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:34.562136 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.562039 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:34.656777 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:34.656742 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:35.076400 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:25:35.076364 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd029bf9f_23a3_47c7_8eca_e343baed7d99.slice/crio-bd7fb9d2ec83abf4fc9425b7d070b8560027ae0059344634f2510a59b868cefd WatchSource:0}: Error finding container bd7fb9d2ec83abf4fc9425b7d070b8560027ae0059344634f2510a59b868cefd: Status 404 returned error can't find the container with id bd7fb9d2ec83abf4fc9425b7d070b8560027ae0059344634f2510a59b868cefd Nov 03 02:25:35.078938 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.078902 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:25:35.079029 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.078975 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:25:35.079029 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.079003 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:25:35.079725 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.079699 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd"] Nov 03 02:25:35.810092 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.810029 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:25:35.812097 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.812070 5463 generic.go:356] "Generic (PLEG): container finished" podID="255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" containerID="fb5c65ed4de6baf0dd3d2cdf6d7f2c33abc9343b79e138f1cfb283fa31329399" exitCode=0 Nov 03 02:25:35.812268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.812115 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" Nov 03 02:25:35.812268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.812112 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" event={"ID":"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e","Type":"ContainerDied","Data":"fb5c65ed4de6baf0dd3d2cdf6d7f2c33abc9343b79e138f1cfb283fa31329399"} Nov 03 02:25:35.812268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.812184 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd" event={"ID":"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e","Type":"ContainerDied","Data":"e49dabc85c928a0e7c58803995609a2e2834459b2a645a89d3261249b7e0bd2a"} Nov 03 02:25:35.812268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.812231 5463 scope.go:122] "RemoveContainer" containerID="fb5c65ed4de6baf0dd3d2cdf6d7f2c33abc9343b79e138f1cfb283fa31329399" Nov 03 02:25:35.814163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.814143 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" event={"ID":"d029bf9f-23a3-47c7-8eca-e343baed7d99","Type":"ContainerStarted","Data":"93e1f9bbae83197112663116fd74171229e18aef93fe1ea898fbd10f5b7296fb"} Nov 03 02:25:35.814265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.814179 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" event={"ID":"d029bf9f-23a3-47c7-8eca-e343baed7d99","Type":"ContainerStarted","Data":"bd7fb9d2ec83abf4fc9425b7d070b8560027ae0059344634f2510a59b868cefd"} Nov 03 02:25:35.823659 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.822921 5463 scope.go:122] "RemoveContainer" containerID="fe60cace8e3b1fbf70992ffacac504bcb16db23fdbe1286a97c2c387bb6b77ae" Nov 03 02:25:35.836891 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.836872 5463 scope.go:122] "RemoveContainer" containerID="fb5c65ed4de6baf0dd3d2cdf6d7f2c33abc9343b79e138f1cfb283fa31329399" Nov 03 02:25:35.842664 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:25:35.837684 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb5c65ed4de6baf0dd3d2cdf6d7f2c33abc9343b79e138f1cfb283fa31329399\": container with ID starting with fb5c65ed4de6baf0dd3d2cdf6d7f2c33abc9343b79e138f1cfb283fa31329399 not found: ID does not exist" containerID="fb5c65ed4de6baf0dd3d2cdf6d7f2c33abc9343b79e138f1cfb283fa31329399" Nov 03 02:25:35.842664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.837749 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb5c65ed4de6baf0dd3d2cdf6d7f2c33abc9343b79e138f1cfb283fa31329399"} err="failed to get container status \"fb5c65ed4de6baf0dd3d2cdf6d7f2c33abc9343b79e138f1cfb283fa31329399\": rpc error: code = NotFound desc = could not find container \"fb5c65ed4de6baf0dd3d2cdf6d7f2c33abc9343b79e138f1cfb283fa31329399\": container with ID starting with fb5c65ed4de6baf0dd3d2cdf6d7f2c33abc9343b79e138f1cfb283fa31329399 not found: ID does not exist" Nov 03 02:25:35.842664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.837774 5463 scope.go:122] "RemoveContainer" containerID="fe60cace8e3b1fbf70992ffacac504bcb16db23fdbe1286a97c2c387bb6b77ae" Nov 03 02:25:35.842664 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:25:35.840031 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe60cace8e3b1fbf70992ffacac504bcb16db23fdbe1286a97c2c387bb6b77ae\": container with ID starting with fe60cace8e3b1fbf70992ffacac504bcb16db23fdbe1286a97c2c387bb6b77ae not found: ID does not exist" containerID="fe60cace8e3b1fbf70992ffacac504bcb16db23fdbe1286a97c2c387bb6b77ae" Nov 03 02:25:35.842664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.840067 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe60cace8e3b1fbf70992ffacac504bcb16db23fdbe1286a97c2c387bb6b77ae"} err="failed to get container status \"fe60cace8e3b1fbf70992ffacac504bcb16db23fdbe1286a97c2c387bb6b77ae\": rpc error: code = NotFound desc = could not find container \"fe60cace8e3b1fbf70992ffacac504bcb16db23fdbe1286a97c2c387bb6b77ae\": container with ID starting with fe60cace8e3b1fbf70992ffacac504bcb16db23fdbe1286a97c2c387bb6b77ae not found: ID does not exist" Nov 03 02:25:35.924631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.924597 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-podinfo\") pod \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " Nov 03 02:25:35.924631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.924636 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-token\" (UniqueName: \"kubernetes.io/projected/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-token\") pod \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " Nov 03 02:25:35.924872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.924745 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-workload-socket\") pod \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " Nov 03 02:25:35.924872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.924787 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-kube-api-access-7gh6b\" (UniqueName: \"kubernetes.io/projected/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-kube-api-access-7gh6b\") pod \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " Nov 03 02:25:35.924872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.924823 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-data\") pod \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " Nov 03 02:25:35.925083 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.924931 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-envoy\") pod \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " Nov 03 02:25:35.925083 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.925005 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-credential-socket\") pod \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " Nov 03 02:25:35.925083 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.925039 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-workload-certs\") pod \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " Nov 03 02:25:35.925283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.925065 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-workload-socket" pod "255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" (UID: "255cfc8e-f969-4678-b7b0-b37a4bc8dd4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:35.925283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.925120 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istiod-ca-cert\") pod \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\" (UID: \"255cfc8e-f969-4678-b7b0-b37a4bc8dd4e\") " Nov 03 02:25:35.925283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.925140 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-data" pod "255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" (UID: "255cfc8e-f969-4678-b7b0-b37a4bc8dd4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:35.925433 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.925340 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-credential-socket" pod "255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" (UID: "255cfc8e-f969-4678-b7b0-b37a4bc8dd4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:35.925433 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.925376 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-workload-certs" pod "255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" (UID: "255cfc8e-f969-4678-b7b0-b37a4bc8dd4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:35.926228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.925729 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istiod-ca-cert" pod "255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" (UID: "255cfc8e-f969-4678-b7b0-b37a4bc8dd4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:25:35.926228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.926112 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:35.926228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.926135 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:35.926228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.926150 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:35.926228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.926164 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:35.926228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.926179 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:35.927164 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.927141 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-token" pod "255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" (UID: "255cfc8e-f969-4678-b7b0-b37a4bc8dd4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:35.927390 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.927362 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-podinfo" pod "255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" (UID: "255cfc8e-f969-4678-b7b0-b37a4bc8dd4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:25:35.927390 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.927386 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-kube-api-access-7gh6b" pod "255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" (UID: "255cfc8e-f969-4678-b7b0-b37a4bc8dd4e"). InnerVolumeSpecName "kube-api-access-7gh6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:35.927524 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:35.927449 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-envoy" pod "255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" (UID: "255cfc8e-f969-4678-b7b0-b37a4bc8dd4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:36.027677 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:36.027648 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gh6b\" (UniqueName: \"kubernetes.io/projected/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-kube-api-access-7gh6b\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:36.027677 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:36.027676 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:36.027870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:36.027686 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:36.027870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:36.027694 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:36.096900 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:36.096792 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" podStartSLOduration=3.096773307 podStartE2EDuration="3.096773307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:25:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:25:36.094129541 +0000 UTC m=+2017.778238696" watchObservedRunningTime="2026-11-03 02:25:36.096773307 +0000 UTC m=+2017.780882462" Nov 03 02:25:36.260016 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:36.259975 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd"] Nov 03 02:25:36.263560 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:36.263535 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad4591-openshift-default-6df5f8d4d5-4fxrd"] Nov 03 02:25:36.444290 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:36.444260 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" path="/var/lib/kubelet/pods/255cfc8e-f969-4678-b7b0-b37a4bc8dd4e/volumes" Nov 03 02:25:36.657420 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:36.657380 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:36.664040 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:36.664007 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:36.823258 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:36.823162 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:36.824009 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:36.823989 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:25:37.958940 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:37.958907 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:38.954596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:38.954565 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:39.117768 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:39.117730 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:39.275383 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:39.275300 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:42.218089 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.218058 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:42.391693 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.391655 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:42.616510 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.615866 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr"] Nov 03 02:25:42.617952 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.617924 5463 cpu_manager.go:414] "containerMap: removing container" podUID="255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" containerName="istio-proxy" Nov 03 02:25:42.617952 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.617949 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" containerName="istio-proxy" Nov 03 02:25:42.618159 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.617966 5463 cpu_manager.go:414] "containerMap: removing container" podUID="255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" containerName="istio-proxy" Nov 03 02:25:42.618159 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.617974 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" containerName="istio-proxy" Nov 03 02:25:42.618159 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.618136 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" containerName="istio-proxy" Nov 03 02:25:42.618159 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.618151 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="255cfc8e-f969-4678-b7b0-b37a4bc8dd4e" containerName="istio-proxy" Nov 03 02:25:42.623566 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.623540 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr" Nov 03 02:25:42.677972 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.677941 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f760d3\"" Nov 03 02:25:42.680913 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.680883 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr"] Nov 03 02:25:42.700915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.700880 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:42.718183 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.718147 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284b69a9-1ced-4c2f-9698-7682e921ca69-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr\" (UID: \"284b69a9-1ced-4c2f-9698-7682e921ca69\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr" Nov 03 02:25:42.718340 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.718324 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8zjg\" (UniqueName: \"kubernetes.io/projected/284b69a9-1ced-4c2f-9698-7682e921ca69-kube-api-access-w8zjg\") pod \"payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr\" (UID: \"284b69a9-1ced-4c2f-9698-7682e921ca69\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr" Nov 03 02:25:42.726966 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.726947 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f760d3-dockercfg-pj7z4\"" Nov 03 02:25:42.820991 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.820954 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8zjg\" (UniqueName: \"kubernetes.io/projected/284b69a9-1ced-4c2f-9698-7682e921ca69-kube-api-access-w8zjg\") pod \"payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr\" (UID: \"284b69a9-1ced-4c2f-9698-7682e921ca69\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr" Nov 03 02:25:42.821350 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.821327 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284b69a9-1ced-4c2f-9698-7682e921ca69-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr\" (UID: \"284b69a9-1ced-4c2f-9698-7682e921ca69\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr" Nov 03 02:25:42.822289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.822264 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284b69a9-1ced-4c2f-9698-7682e921ca69-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr\" (UID: \"284b69a9-1ced-4c2f-9698-7682e921ca69\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr" Nov 03 02:25:42.918209 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.912114 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8zjg\" (UniqueName: \"kubernetes.io/projected/284b69a9-1ced-4c2f-9698-7682e921ca69-kube-api-access-w8zjg\") pod \"payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr\" (UID: \"284b69a9-1ced-4c2f-9698-7682e921ca69\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr" Nov 03 02:25:42.942426 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:42.942384 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr" Nov 03 02:25:43.124435 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:25:43.124397 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:25:43.402040 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:25:43.402011 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod284b69a9_1ced_4c2f_9698_7682e921ca69.slice/crio-e305d04a2c5eb4885ba5e281a7a9915942338eb2df8bba8aba364a9cff5fb8b2 WatchSource:0}: Error finding container e305d04a2c5eb4885ba5e281a7a9915942338eb2df8bba8aba364a9cff5fb8b2: Status 404 returned error can't find the container with id e305d04a2c5eb4885ba5e281a7a9915942338eb2df8bba8aba364a9cff5fb8b2 Nov 03 02:25:43.404164 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:43.404138 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr"] Nov 03 02:25:43.909171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:43.909132 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr" event={"ID":"284b69a9-1ced-4c2f-9698-7682e921ca69","Type":"ContainerStarted","Data":"53e11a75a19c1ee73028ed559c380ea2165876775d1d42736df5b512ba240a6a"} Nov 03 02:25:43.909343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:43.909221 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr" event={"ID":"284b69a9-1ced-4c2f-9698-7682e921ca69","Type":"ContainerStarted","Data":"e305d04a2c5eb4885ba5e281a7a9915942338eb2df8bba8aba364a9cff5fb8b2"} Nov 03 02:25:43.909400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:43.909337 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr" Nov 03 02:25:44.025672 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:44.025619 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr" podStartSLOduration=2.025605792 podStartE2EDuration="2.025605792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:25:44.022485188 +0000 UTC m=+2025.706594347" watchObservedRunningTime="2026-11-03 02:25:44.025605792 +0000 UTC m=+2025.709714946" Nov 03 02:25:44.515362 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:44.515325 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:46.174927 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:46.174894 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:47.409463 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:47.409428 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:47.960721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:47.960684 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz"] Nov 03 02:25:47.966242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:47.966219 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:47.996212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:47.989154 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbecd0cc-1384-46b8-8eaa-758627373292-istiod-ca-cert\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:47.996212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:47.989230 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-istio-envoy\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:47.996212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:47.989261 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbecd0cc-1384-46b8-8eaa-758627373292-istio-podinfo\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:47.996212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:47.989298 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-workload-socket\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:47.996212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:47.989466 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbecd0cc-1384-46b8-8eaa-758627373292-istio-token\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:47.996212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:47.989545 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-workload-certs\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:47.996212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:47.989569 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76j7f\" (UniqueName: \"kubernetes.io/projected/cbecd0cc-1384-46b8-8eaa-758627373292-kube-api-access-76j7f\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:47.996212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:47.989650 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-credential-socket\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:47.996212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:47.989675 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-istio-data\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:48.057533 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.057035 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-7f9d9d-openshift-default-dockercfg-54rkr\"" Nov 03 02:25:48.075951 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.073340 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:48.086696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.086664 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz"] Nov 03 02:25:48.093038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.091865 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbecd0cc-1384-46b8-8eaa-758627373292-istiod-ca-cert\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:48.093038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.091915 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-istio-envoy\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:48.093038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.091949 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbecd0cc-1384-46b8-8eaa-758627373292-istio-podinfo\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:48.093038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.091990 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-workload-socket\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:48.093038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.092123 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbecd0cc-1384-46b8-8eaa-758627373292-istio-token\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:48.093038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.092225 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-workload-certs\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:48.093038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.092287 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76j7f\" (UniqueName: \"kubernetes.io/projected/cbecd0cc-1384-46b8-8eaa-758627373292-kube-api-access-76j7f\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:48.093038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.092386 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-credential-socket\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:48.093038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.092416 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-istio-data\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:48.093038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.092911 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-istio-data\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:48.095114 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.095056 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-credential-socket\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:48.095947 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.095902 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbecd0cc-1384-46b8-8eaa-758627373292-istiod-ca-cert\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:48.097473 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.097421 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-workload-certs\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:48.097898 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.097856 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-workload-socket\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:48.122858 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.118269 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-istio-envoy\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:48.123790 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.123772 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbecd0cc-1384-46b8-8eaa-758627373292-istio-podinfo\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:48.150757 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.149324 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbecd0cc-1384-46b8-8eaa-758627373292-istio-token\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:48.153529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.153494 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76j7f\" (UniqueName: \"kubernetes.io/projected/cbecd0cc-1384-46b8-8eaa-758627373292-kube-api-access-76j7f\") pod \"e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:48.282728 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.282641 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:48.391966 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.388295 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:48.735634 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.735570 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz"] Nov 03 02:25:48.739481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.738718 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:25:48.739481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.738828 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:25:48.739481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.738870 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:25:48.952073 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.952029 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" event={"ID":"cbecd0cc-1384-46b8-8eaa-758627373292","Type":"ContainerStarted","Data":"fd2edee22c38d95c9e6e3273e986c4ecd2c743c1efeb7c94a413853922eb347c"} Nov 03 02:25:48.952310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:48.952133 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" event={"ID":"cbecd0cc-1384-46b8-8eaa-758627373292","Type":"ContainerStarted","Data":"c4d58b94df91c19f42126193a888fd749f3c4dc1912c3979a866d81a55dc5b96"} Nov 03 02:25:49.070304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:49.070201 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" podStartSLOduration=2.070169528 podStartE2EDuration="2.070169528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:25:49.068205187 +0000 UTC m=+2030.752314343" watchObservedRunningTime="2026-11-03 02:25:49.070169528 +0000 UTC m=+2030.754278684" Nov 03 02:25:49.283396 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:49.283351 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:49.289263 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:49.289235 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:49.563215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:49.555719 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:49.717913 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:49.717879 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:49.877482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:49.877391 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:49.979822 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:49.979787 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:49.980805 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:49.980787 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:25:51.019263 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:51.019228 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:51.520405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:51.519830 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz"] Nov 03 02:25:51.520405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:51.520124 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" podUID="871baea4-93af-43d2-8ef7-0b05c4f50526" containerName="istio-proxy" containerID="cri-o://32c3825f11d1e75f6c0c9e054a0fffb4805429db3f9434bd87b75794ab07c0f3" gracePeriod=30 Nov 03 02:25:51.532703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:51.532681 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:52.342877 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:52.342845 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:52.766213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:52.765571 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:52.988790 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:52.988744 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:53.145651 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:53.145618 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:54.920254 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:54.920183 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr" Nov 03 02:25:55.041350 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:55.041266 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:56.004371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:56.004329 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:56.182289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:56.182005 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:56.878056 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:56.878028 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:25:56.994713 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:56.994675 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-envoy\") pod \"871baea4-93af-43d2-8ef7-0b05c4f50526\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " Nov 03 02:25:56.994931 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:56.994770 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-token\" (UniqueName: \"kubernetes.io/projected/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-token\") pod \"871baea4-93af-43d2-8ef7-0b05c4f50526\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " Nov 03 02:25:56.994931 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:56.994807 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-workload-socket\") pod \"871baea4-93af-43d2-8ef7-0b05c4f50526\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " Nov 03 02:25:56.994931 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:56.994845 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-podinfo\") pod \"871baea4-93af-43d2-8ef7-0b05c4f50526\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " Nov 03 02:25:56.994931 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:56.994887 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-credential-socket\") pod \"871baea4-93af-43d2-8ef7-0b05c4f50526\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " Nov 03 02:25:56.995157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:56.994981 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/871baea4-93af-43d2-8ef7-0b05c4f50526-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871baea4-93af-43d2-8ef7-0b05c4f50526-istiod-ca-cert\") pod \"871baea4-93af-43d2-8ef7-0b05c4f50526\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " Nov 03 02:25:56.995157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:56.995043 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-data\") pod \"871baea4-93af-43d2-8ef7-0b05c4f50526\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " Nov 03 02:25:56.995157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:56.995147 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/871baea4-93af-43d2-8ef7-0b05c4f50526-kube-api-access-57gq6\" (UniqueName: \"kubernetes.io/projected/871baea4-93af-43d2-8ef7-0b05c4f50526-kube-api-access-57gq6\") pod \"871baea4-93af-43d2-8ef7-0b05c4f50526\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " Nov 03 02:25:56.995401 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:56.995177 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-workload-certs\") pod \"871baea4-93af-43d2-8ef7-0b05c4f50526\" (UID: \"871baea4-93af-43d2-8ef7-0b05c4f50526\") " Nov 03 02:25:56.996258 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:56.995987 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-workload-certs" pod "871baea4-93af-43d2-8ef7-0b05c4f50526" (UID: "871baea4-93af-43d2-8ef7-0b05c4f50526"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:56.996495 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:56.996465 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-credential-socket" pod "871baea4-93af-43d2-8ef7-0b05c4f50526" (UID: "871baea4-93af-43d2-8ef7-0b05c4f50526"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:56.996646 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:56.996622 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-workload-socket" pod "871baea4-93af-43d2-8ef7-0b05c4f50526" (UID: "871baea4-93af-43d2-8ef7-0b05c4f50526"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:56.997178 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:56.997152 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/871baea4-93af-43d2-8ef7-0b05c4f50526-istiod-ca-cert" pod "871baea4-93af-43d2-8ef7-0b05c4f50526" (UID: "871baea4-93af-43d2-8ef7-0b05c4f50526"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:25:56.997468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:56.997446 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-data" pod "871baea4-93af-43d2-8ef7-0b05c4f50526" (UID: "871baea4-93af-43d2-8ef7-0b05c4f50526"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:56.999700 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:56.999660 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871baea4-93af-43d2-8ef7-0b05c4f50526-kube-api-access-57gq6" pod "871baea4-93af-43d2-8ef7-0b05c4f50526" (UID: "871baea4-93af-43d2-8ef7-0b05c4f50526"). InnerVolumeSpecName "kube-api-access-57gq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:57.000267 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.000242 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-envoy" pod "871baea4-93af-43d2-8ef7-0b05c4f50526" (UID: "871baea4-93af-43d2-8ef7-0b05c4f50526"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:57.000475 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.000457 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-token" pod "871baea4-93af-43d2-8ef7-0b05c4f50526" (UID: "871baea4-93af-43d2-8ef7-0b05c4f50526"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:57.003274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.003252 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-podinfo" pod "871baea4-93af-43d2-8ef7-0b05c4f50526" (UID: "871baea4-93af-43d2-8ef7-0b05c4f50526"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:25:57.065235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.057750 5463 generic.go:356] "Generic (PLEG): container finished" podID="871baea4-93af-43d2-8ef7-0b05c4f50526" containerID="32c3825f11d1e75f6c0c9e054a0fffb4805429db3f9434bd87b75794ab07c0f3" exitCode=0 Nov 03 02:25:57.065235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.057858 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" event={"ID":"871baea4-93af-43d2-8ef7-0b05c4f50526","Type":"ContainerDied","Data":"32c3825f11d1e75f6c0c9e054a0fffb4805429db3f9434bd87b75794ab07c0f3"} Nov 03 02:25:57.065235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.057941 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" event={"ID":"871baea4-93af-43d2-8ef7-0b05c4f50526","Type":"ContainerDied","Data":"6cd63137b1832d91c14adfcdf6d52898179d225d3b91ed1b5bf54e274c094401"} Nov 03 02:25:57.065235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.057982 5463 scope.go:122] "RemoveContainer" containerID="32c3825f11d1e75f6c0c9e054a0fffb4805429db3f9434bd87b75794ab07c0f3" Nov 03 02:25:57.065235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.058228 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz" Nov 03 02:25:57.085176 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.085152 5463 scope.go:122] "RemoveContainer" containerID="58896957410a6ed50693266f6766b0bf5a3019aa55b56c1fbc1a520a40c4effb" Nov 03 02:25:57.097115 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.097045 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:57.097115 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.097085 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:57.097115 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.097101 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:57.097385 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.097116 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:57.097385 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.097131 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:57.097385 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.097144 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/871baea4-93af-43d2-8ef7-0b05c4f50526-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:57.097385 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.097158 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:57.097385 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.097217 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57gq6\" (UniqueName: \"kubernetes.io/projected/871baea4-93af-43d2-8ef7-0b05c4f50526-kube-api-access-57gq6\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:57.097385 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.097235 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/871baea4-93af-43d2-8ef7-0b05c4f50526-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:57.105020 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.104991 5463 scope.go:122] "RemoveContainer" containerID="32c3825f11d1e75f6c0c9e054a0fffb4805429db3f9434bd87b75794ab07c0f3" Nov 03 02:25:57.105437 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:25:57.105410 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c3825f11d1e75f6c0c9e054a0fffb4805429db3f9434bd87b75794ab07c0f3\": container with ID starting with 32c3825f11d1e75f6c0c9e054a0fffb4805429db3f9434bd87b75794ab07c0f3 not found: ID does not exist" containerID="32c3825f11d1e75f6c0c9e054a0fffb4805429db3f9434bd87b75794ab07c0f3" Nov 03 02:25:57.105627 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.105597 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c3825f11d1e75f6c0c9e054a0fffb4805429db3f9434bd87b75794ab07c0f3"} err="failed to get container status \"32c3825f11d1e75f6c0c9e054a0fffb4805429db3f9434bd87b75794ab07c0f3\": rpc error: code = NotFound desc = could not find container \"32c3825f11d1e75f6c0c9e054a0fffb4805429db3f9434bd87b75794ab07c0f3\": container with ID starting with 32c3825f11d1e75f6c0c9e054a0fffb4805429db3f9434bd87b75794ab07c0f3 not found: ID does not exist" Nov 03 02:25:57.105770 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.105756 5463 scope.go:122] "RemoveContainer" containerID="58896957410a6ed50693266f6766b0bf5a3019aa55b56c1fbc1a520a40c4effb" Nov 03 02:25:57.106627 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:25:57.106601 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58896957410a6ed50693266f6766b0bf5a3019aa55b56c1fbc1a520a40c4effb\": container with ID starting with 58896957410a6ed50693266f6766b0bf5a3019aa55b56c1fbc1a520a40c4effb not found: ID does not exist" containerID="58896957410a6ed50693266f6766b0bf5a3019aa55b56c1fbc1a520a40c4effb" Nov 03 02:25:57.106801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.106774 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58896957410a6ed50693266f6766b0bf5a3019aa55b56c1fbc1a520a40c4effb"} err="failed to get container status \"58896957410a6ed50693266f6766b0bf5a3019aa55b56c1fbc1a520a40c4effb\": rpc error: code = NotFound desc = could not find container \"58896957410a6ed50693266f6766b0bf5a3019aa55b56c1fbc1a520a40c4effb\": container with ID starting with 58896957410a6ed50693266f6766b0bf5a3019aa55b56c1fbc1a520a40c4effb not found: ID does not exist" Nov 03 02:25:57.235729 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.235695 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz"] Nov 03 02:25:57.239676 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:57.239647 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3bbfc4-openshift-default-7b8bbb496f-jvmdz"] Nov 03 02:25:58.445295 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:58.445231 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="871baea4-93af-43d2-8ef7-0b05c4f50526" path="/var/lib/kubelet/pods/871baea4-93af-43d2-8ef7-0b05c4f50526/volumes" Nov 03 02:25:58.505330 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:58.505258 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:59.076112 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:59.076077 5463 generic.go:356] "Generic (PLEG): container finished" podID="17c81363-4f1f-4550-ba15-35f9e86eccf7" containerID="84400737403c98c63902b2439174cd25ae46e5336a60892423e2b597ebf71be0" exitCode=0 Nov 03 02:25:59.076292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:59.076147 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" event={"ID":"17c81363-4f1f-4550-ba15-35f9e86eccf7","Type":"ContainerDied","Data":"84400737403c98c63902b2439174cd25ae46e5336a60892423e2b597ebf71be0"} Nov 03 02:25:59.076566 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:59.076548 5463 scope.go:122] "RemoveContainer" containerID="84400737403c98c63902b2439174cd25ae46e5336a60892423e2b597ebf71be0" Nov 03 02:25:59.077934 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:59.077907 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:25:59.077984 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:59.077976 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:25:59.078034 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:59.078014 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:25:59.902651 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:25:59.902614 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:00.071215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:00.069788 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:00.091322 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:00.091283 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" event={"ID":"17c81363-4f1f-4550-ba15-35f9e86eccf7","Type":"ContainerStarted","Data":"da4f47d76050ddbe3590a7f86e4ea9ea6c52cdfb7ae19d25894feaf30463371f"} Nov 03 02:26:00.379730 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:00.379639 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:00.874211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:00.873465 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:26:00.878100 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:00.878067 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 02:26:00.878290 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:00.878119 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" podUID="17c81363-4f1f-4550-ba15-35f9e86eccf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 02:26:01.872582 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:01.872542 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 02:26:01.873115 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:01.872609 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" podUID="17c81363-4f1f-4550-ba15-35f9e86eccf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 02:26:02.874947 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:02.874507 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 02:26:02.874947 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:02.874580 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" podUID="17c81363-4f1f-4550-ba15-35f9e86eccf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 02:26:03.875556 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:03.875519 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 02:26:03.876181 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:03.876148 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" podUID="17c81363-4f1f-4550-ba15-35f9e86eccf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 02:26:04.011652 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:04.011616 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:04.823628 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:04.823539 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:26:04.878639 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:04.878606 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 02:26:04.879148 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:04.878819 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" podUID="17c81363-4f1f-4550-ba15-35f9e86eccf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 02:26:05.877293 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:05.877262 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:26:05.878472 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:05.878447 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:26:07.008463 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:07.008427 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:11.698645 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:11.698603 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:13.007149 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:26:13.007113 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:26:14.621809 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:14.621772 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:16.088756 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.088719 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659"] Nov 03 02:26:16.089325 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.088930 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659" podUID="98497121-d021-473e-83f8-fe5e3c3101ac" containerName="payload-processing" containerID="cri-o://cbc0efd7ecd027d355ead1f383dd1c01d407185ec94810a1b13541174f5d8172" gracePeriod=30 Nov 03 02:26:16.116859 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.116834 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:16.144378 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.144347 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75"] Nov 03 02:26:16.144559 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.144538 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75" podUID="2e20d8b1-8b29-4d61-a854-071821501552" containerName="payload-pre-processing" containerID="cri-o://54f6d5fb3064bce8e875315a97ae9a45b96063b630a97e3f04203a0551bd03f5" gracePeriod=30 Nov 03 02:26:16.223296 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.223265 5463 generic.go:356] "Generic (PLEG): container finished" podID="98497121-d021-473e-83f8-fe5e3c3101ac" containerID="cbc0efd7ecd027d355ead1f383dd1c01d407185ec94810a1b13541174f5d8172" exitCode=0 Nov 03 02:26:16.223447 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.223306 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659" event={"ID":"98497121-d021-473e-83f8-fe5e3c3101ac","Type":"ContainerDied","Data":"cbc0efd7ecd027d355ead1f383dd1c01d407185ec94810a1b13541174f5d8172"} Nov 03 02:26:16.271560 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.271525 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:16.424336 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.424282 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:16.434202 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.434167 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659" Nov 03 02:26:16.458148 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.458126 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75" Nov 03 02:26:16.578747 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.578704 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98497121-d021-473e-83f8-fe5e3c3101ac-kube-api-access-xvbkq\" (UniqueName: \"kubernetes.io/projected/98497121-d021-473e-83f8-fe5e3c3101ac-kube-api-access-xvbkq\") pod \"98497121-d021-473e-83f8-fe5e3c3101ac\" (UID: \"98497121-d021-473e-83f8-fe5e3c3101ac\") " Nov 03 02:26:16.578920 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.578776 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98497121-d021-473e-83f8-fe5e3c3101ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98497121-d021-473e-83f8-fe5e3c3101ac-plugins-config-volume\") pod \"98497121-d021-473e-83f8-fe5e3c3101ac\" (UID: \"98497121-d021-473e-83f8-fe5e3c3101ac\") " Nov 03 02:26:16.578920 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.578876 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e20d8b1-8b29-4d61-a854-071821501552-kube-api-access-8xwlv\" (UniqueName: \"kubernetes.io/projected/2e20d8b1-8b29-4d61-a854-071821501552-kube-api-access-8xwlv\") pod \"2e20d8b1-8b29-4d61-a854-071821501552\" (UID: \"2e20d8b1-8b29-4d61-a854-071821501552\") " Nov 03 02:26:16.578920 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.578910 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e20d8b1-8b29-4d61-a854-071821501552-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e20d8b1-8b29-4d61-a854-071821501552-plugins-config-volume\") pod \"2e20d8b1-8b29-4d61-a854-071821501552\" (UID: \"2e20d8b1-8b29-4d61-a854-071821501552\") " Nov 03 02:26:16.579392 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.579312 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98497121-d021-473e-83f8-fe5e3c3101ac-plugins-config-volume" pod "98497121-d021-473e-83f8-fe5e3c3101ac" (UID: "98497121-d021-473e-83f8-fe5e3c3101ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:26:16.579392 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.579330 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e20d8b1-8b29-4d61-a854-071821501552-plugins-config-volume" pod "2e20d8b1-8b29-4d61-a854-071821501552" (UID: "2e20d8b1-8b29-4d61-a854-071821501552"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:26:16.579567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.579511 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98497121-d021-473e-83f8-fe5e3c3101ac-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:16.579567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.579534 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e20d8b1-8b29-4d61-a854-071821501552-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:16.581517 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.581458 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e20d8b1-8b29-4d61-a854-071821501552-kube-api-access-8xwlv" pod "2e20d8b1-8b29-4d61-a854-071821501552" (UID: "2e20d8b1-8b29-4d61-a854-071821501552"). InnerVolumeSpecName "kube-api-access-8xwlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:26:16.581918 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.581887 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98497121-d021-473e-83f8-fe5e3c3101ac-kube-api-access-xvbkq" pod "98497121-d021-473e-83f8-fe5e3c3101ac" (UID: "98497121-d021-473e-83f8-fe5e3c3101ac"). InnerVolumeSpecName "kube-api-access-xvbkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:26:16.681111 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.681062 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvbkq\" (UniqueName: \"kubernetes.io/projected/98497121-d021-473e-83f8-fe5e3c3101ac-kube-api-access-xvbkq\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:16.681111 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:16.681102 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xwlv\" (UniqueName: \"kubernetes.io/projected/2e20d8b1-8b29-4d61-a854-071821501552-kube-api-access-8xwlv\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:17.231468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:17.231432 5463 generic.go:356] "Generic (PLEG): container finished" podID="2e20d8b1-8b29-4d61-a854-071821501552" containerID="54f6d5fb3064bce8e875315a97ae9a45b96063b630a97e3f04203a0551bd03f5" exitCode=0 Nov 03 02:26:17.231962 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:17.231493 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75" Nov 03 02:26:17.231962 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:17.231516 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75" event={"ID":"2e20d8b1-8b29-4d61-a854-071821501552","Type":"ContainerDied","Data":"54f6d5fb3064bce8e875315a97ae9a45b96063b630a97e3f04203a0551bd03f5"} Nov 03 02:26:17.231962 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:17.231584 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75" event={"ID":"2e20d8b1-8b29-4d61-a854-071821501552","Type":"ContainerDied","Data":"ac99dd851886be7779b0ac8f068fb3722288d5476ac3bdb63f2ff159b3f3e26b"} Nov 03 02:26:17.231962 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:17.231613 5463 scope.go:122] "RemoveContainer" containerID="54f6d5fb3064bce8e875315a97ae9a45b96063b630a97e3f04203a0551bd03f5" Nov 03 02:26:17.233362 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:17.233334 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659" event={"ID":"98497121-d021-473e-83f8-fe5e3c3101ac","Type":"ContainerDied","Data":"a0d385c3f7f228d0552117a06b1c07c5fea25e97a4dc9ded6fdd48868fc0cbfa"} Nov 03 02:26:17.233566 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:17.233342 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659" Nov 03 02:26:17.243603 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:17.243580 5463 scope.go:122] "RemoveContainer" containerID="54f6d5fb3064bce8e875315a97ae9a45b96063b630a97e3f04203a0551bd03f5" Nov 03 02:26:17.244033 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:26:17.243924 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f6d5fb3064bce8e875315a97ae9a45b96063b630a97e3f04203a0551bd03f5\": container with ID starting with 54f6d5fb3064bce8e875315a97ae9a45b96063b630a97e3f04203a0551bd03f5 not found: ID does not exist" containerID="54f6d5fb3064bce8e875315a97ae9a45b96063b630a97e3f04203a0551bd03f5" Nov 03 02:26:17.244033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:17.243956 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f6d5fb3064bce8e875315a97ae9a45b96063b630a97e3f04203a0551bd03f5"} err="failed to get container status \"54f6d5fb3064bce8e875315a97ae9a45b96063b630a97e3f04203a0551bd03f5\": rpc error: code = NotFound desc = could not find container \"54f6d5fb3064bce8e875315a97ae9a45b96063b630a97e3f04203a0551bd03f5\": container with ID starting with 54f6d5fb3064bce8e875315a97ae9a45b96063b630a97e3f04203a0551bd03f5 not found: ID does not exist" Nov 03 02:26:17.244033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:17.243973 5463 scope.go:122] "RemoveContainer" containerID="cbc0efd7ecd027d355ead1f383dd1c01d407185ec94810a1b13541174f5d8172" Nov 03 02:26:17.397512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:17.397480 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75"] Nov 03 02:26:17.402174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:17.402145 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9bd3ab-7756cd4b67-pkq75"] Nov 03 02:26:17.511747 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:17.511716 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659"] Nov 03 02:26:17.514825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:17.514804 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9bd3ab-78cc8c8598-5q659"] Nov 03 02:26:17.592562 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:17.592527 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:18.446304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:18.446272 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e20d8b1-8b29-4d61-a854-071821501552" path="/var/lib/kubelet/pods/2e20d8b1-8b29-4d61-a854-071821501552/volumes" Nov 03 02:26:18.446759 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:18.446670 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98497121-d021-473e-83f8-fe5e3c3101ac" path="/var/lib/kubelet/pods/98497121-d021-473e-83f8-fe5e3c3101ac/volumes" Nov 03 02:26:19.716704 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:19.716655 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:20.286935 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:20.286900 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq"] Nov 03 02:26:20.393591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:20.393557 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc"] Nov 03 02:26:20.681915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:20.681852 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:20.840698 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:20.840656 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:22.447697 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:22.447662 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:24.443798 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.443765 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm"] Nov 03 02:26:24.444539 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.444516 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2e20d8b1-8b29-4d61-a854-071821501552" containerName="payload-pre-processing" Nov 03 02:26:24.444649 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.444539 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e20d8b1-8b29-4d61-a854-071821501552" containerName="payload-pre-processing" Nov 03 02:26:24.444649 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.444573 5463 cpu_manager.go:414] "containerMap: removing container" podUID="871baea4-93af-43d2-8ef7-0b05c4f50526" containerName="istio-proxy" Nov 03 02:26:24.444649 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.444581 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="871baea4-93af-43d2-8ef7-0b05c4f50526" containerName="istio-proxy" Nov 03 02:26:24.444649 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.444596 5463 cpu_manager.go:414] "containerMap: removing container" podUID="98497121-d021-473e-83f8-fe5e3c3101ac" containerName="payload-processing" Nov 03 02:26:24.444649 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.444603 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98497121-d021-473e-83f8-fe5e3c3101ac" containerName="payload-processing" Nov 03 02:26:24.444914 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.444784 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="871baea4-93af-43d2-8ef7-0b05c4f50526" containerName="istio-proxy" Nov 03 02:26:24.444914 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.444799 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="98497121-d021-473e-83f8-fe5e3c3101ac" containerName="payload-processing" Nov 03 02:26:24.444914 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.444822 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e20d8b1-8b29-4d61-a854-071821501552" containerName="payload-pre-processing" Nov 03 02:26:24.448595 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.448571 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:24.482320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.482263 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-home\") pod \"ipp-route-56299c-kserve-976d85fc-fl7fm\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:24.482439 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.482378 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa346df3-a524-419b-b3a4-abf4ced623c0-tls-certs\") pod \"ipp-route-56299c-kserve-976d85fc-fl7fm\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:24.482490 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.482461 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-tmp-dir\") pod \"ipp-route-56299c-kserve-976d85fc-fl7fm\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:24.482573 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.482553 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-dshm\") pod \"ipp-route-56299c-kserve-976d85fc-fl7fm\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:24.482831 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.482810 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-model-cache\") pod \"ipp-route-56299c-kserve-976d85fc-fl7fm\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:24.482929 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.482859 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sqnm\" (UniqueName: \"kubernetes.io/projected/aa346df3-a524-419b-b3a4-abf4ced623c0-kube-api-access-7sqnm\") pod \"ipp-route-56299c-kserve-976d85fc-fl7fm\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:24.496987 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.496955 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm"] Nov 03 02:26:24.543735 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.543703 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-56299c\"/\"kube-root-ca.crt\"" Nov 03 02:26:24.543735 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.543713 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-56299c\"/\"default-dockercfg-mmpkl\"" Nov 03 02:26:24.543903 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.543745 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-56299c\"/\"openshift-service-ca.crt\"" Nov 03 02:26:24.551383 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.551359 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-56299c\"/\"ipp-route-56299c-kserve-self-signed-certs\"" Nov 03 02:26:24.555250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.555224 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:24.584445 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.584406 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-home\") pod \"ipp-route-56299c-kserve-976d85fc-fl7fm\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:24.584623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.584486 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa346df3-a524-419b-b3a4-abf4ced623c0-tls-certs\") pod \"ipp-route-56299c-kserve-976d85fc-fl7fm\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:24.584623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.584544 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-tmp-dir\") pod \"ipp-route-56299c-kserve-976d85fc-fl7fm\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:24.584738 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.584671 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-dshm\") pod \"ipp-route-56299c-kserve-976d85fc-fl7fm\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:24.584801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.584743 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-model-cache\") pod \"ipp-route-56299c-kserve-976d85fc-fl7fm\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:24.584801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.584790 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sqnm\" (UniqueName: \"kubernetes.io/projected/aa346df3-a524-419b-b3a4-abf4ced623c0-kube-api-access-7sqnm\") pod \"ipp-route-56299c-kserve-976d85fc-fl7fm\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:24.584905 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.584864 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-home\") pod \"ipp-route-56299c-kserve-976d85fc-fl7fm\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:24.585349 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.585315 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-tmp-dir\") pod \"ipp-route-56299c-kserve-976d85fc-fl7fm\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:24.585477 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.585404 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-model-cache\") pod \"ipp-route-56299c-kserve-976d85fc-fl7fm\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:24.587145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.587120 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-dshm\") pod \"ipp-route-56299c-kserve-976d85fc-fl7fm\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:24.587614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.587594 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa346df3-a524-419b-b3a4-abf4ced623c0-tls-certs\") pod \"ipp-route-56299c-kserve-976d85fc-fl7fm\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:24.644276 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.644244 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sqnm\" (UniqueName: \"kubernetes.io/projected/aa346df3-a524-419b-b3a4-abf4ced623c0-kube-api-access-7sqnm\") pod \"ipp-route-56299c-kserve-976d85fc-fl7fm\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:24.761518 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:24.761437 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:25.147012 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:26:25.146976 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa346df3_a524_419b_b3a4_abf4ced623c0.slice/crio-34c9bdf7b82676f02b198d82951c16df2b2488d2cf8f961e573a1aa31d3feb80 WatchSource:0}: Error finding container 34c9bdf7b82676f02b198d82951c16df2b2488d2cf8f961e573a1aa31d3feb80: Status 404 returned error can't find the container with id 34c9bdf7b82676f02b198d82951c16df2b2488d2cf8f961e573a1aa31d3feb80 Nov 03 02:26:25.149651 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.149622 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm"] Nov 03 02:26:25.287750 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.287673 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq" podUID="2b8b53d3-cac3-4296-a612-e4f4604ce041" containerName="payload-processing" containerID="cri-o://31b46eb25f4d08c2f31864aedc135b09b8e04ea16f0732dfe287dbbcf94d02d6" gracePeriod=25 Nov 03 02:26:25.298703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.298679 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" event={"ID":"aa346df3-a524-419b-b3a4-abf4ced623c0","Type":"ContainerStarted","Data":"fae2b633443e001b21d5f313be963a91102dc7a46c22352077a83f4360fc72cd"} Nov 03 02:26:25.298873 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.298852 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" event={"ID":"aa346df3-a524-419b-b3a4-abf4ced623c0","Type":"ContainerStarted","Data":"34c9bdf7b82676f02b198d82951c16df2b2488d2cf8f961e573a1aa31d3feb80"} Nov 03 02:26:25.394655 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.394608 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc" podUID="b7c9cf2a-a41a-45c3-897e-140344ea67ff" containerName="payload-pre-processing" containerID="cri-o://ec5271a67eb644d44b7dde24a129d279130d039d0d2b7b35674916ae3abc4ec1" gracePeriod=25 Nov 03 02:26:25.415038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.414991 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" podStartSLOduration=1.414977953 podStartE2EDuration="1.414977953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:26:25.411704486 +0000 UTC m=+2067.095813641" watchObservedRunningTime="2026-11-03 02:26:25.414977953 +0000 UTC m=+2067.099087109" Nov 03 02:26:25.562331 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.562305 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq" Nov 03 02:26:25.603847 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.603766 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b8b53d3-cac3-4296-a612-e4f4604ce041-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b8b53d3-cac3-4296-a612-e4f4604ce041-plugins-config-volume\") pod \"2b8b53d3-cac3-4296-a612-e4f4604ce041\" (UID: \"2b8b53d3-cac3-4296-a612-e4f4604ce041\") " Nov 03 02:26:25.603847 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.603808 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b8b53d3-cac3-4296-a612-e4f4604ce041-kube-api-access-ck68n\" (UniqueName: \"kubernetes.io/projected/2b8b53d3-cac3-4296-a612-e4f4604ce041-kube-api-access-ck68n\") pod \"2b8b53d3-cac3-4296-a612-e4f4604ce041\" (UID: \"2b8b53d3-cac3-4296-a612-e4f4604ce041\") " Nov 03 02:26:25.604903 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.604875 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b8b53d3-cac3-4296-a612-e4f4604ce041-plugins-config-volume" pod "2b8b53d3-cac3-4296-a612-e4f4604ce041" (UID: "2b8b53d3-cac3-4296-a612-e4f4604ce041"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:26:25.607486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.607462 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b8b53d3-cac3-4296-a612-e4f4604ce041-kube-api-access-ck68n" pod "2b8b53d3-cac3-4296-a612-e4f4604ce041" (UID: "2b8b53d3-cac3-4296-a612-e4f4604ce041"). InnerVolumeSpecName "kube-api-access-ck68n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:26:25.705775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.705746 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b8b53d3-cac3-4296-a612-e4f4604ce041-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:25.705775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.705775 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck68n\" (UniqueName: \"kubernetes.io/projected/2b8b53d3-cac3-4296-a612-e4f4604ce041-kube-api-access-ck68n\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:25.719408 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.719384 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc" Nov 03 02:26:25.762154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.762120 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:25.807054 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.807029 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7c9cf2a-a41a-45c3-897e-140344ea67ff-kube-api-access-pjffp\" (UniqueName: \"kubernetes.io/projected/b7c9cf2a-a41a-45c3-897e-140344ea67ff-kube-api-access-pjffp\") pod \"b7c9cf2a-a41a-45c3-897e-140344ea67ff\" (UID: \"b7c9cf2a-a41a-45c3-897e-140344ea67ff\") " Nov 03 02:26:25.807219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.807062 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7c9cf2a-a41a-45c3-897e-140344ea67ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c9cf2a-a41a-45c3-897e-140344ea67ff-plugins-config-volume\") pod \"b7c9cf2a-a41a-45c3-897e-140344ea67ff\" (UID: \"b7c9cf2a-a41a-45c3-897e-140344ea67ff\") " Nov 03 02:26:25.807539 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.807515 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c9cf2a-a41a-45c3-897e-140344ea67ff-plugins-config-volume" pod "b7c9cf2a-a41a-45c3-897e-140344ea67ff" (UID: "b7c9cf2a-a41a-45c3-897e-140344ea67ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:26:25.809768 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.809740 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c9cf2a-a41a-45c3-897e-140344ea67ff-kube-api-access-pjffp" pod "b7c9cf2a-a41a-45c3-897e-140344ea67ff" (UID: "b7c9cf2a-a41a-45c3-897e-140344ea67ff"). InnerVolumeSpecName "kube-api-access-pjffp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:26:25.827155 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.827124 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:25.908861 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.908827 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjffp\" (UniqueName: \"kubernetes.io/projected/b7c9cf2a-a41a-45c3-897e-140344ea67ff-kube-api-access-pjffp\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:25.908861 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.908856 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c9cf2a-a41a-45c3-897e-140344ea67ff-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:25.973787 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:25.973692 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:26.306696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:26.306617 5463 generic.go:356] "Generic (PLEG): container finished" podID="b7c9cf2a-a41a-45c3-897e-140344ea67ff" containerID="ec5271a67eb644d44b7dde24a129d279130d039d0d2b7b35674916ae3abc4ec1" exitCode=0 Nov 03 02:26:26.306696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:26.306673 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc" Nov 03 02:26:26.306952 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:26.306708 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc" event={"ID":"b7c9cf2a-a41a-45c3-897e-140344ea67ff","Type":"ContainerDied","Data":"ec5271a67eb644d44b7dde24a129d279130d039d0d2b7b35674916ae3abc4ec1"} Nov 03 02:26:26.306952 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:26.306772 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc" event={"ID":"b7c9cf2a-a41a-45c3-897e-140344ea67ff","Type":"ContainerDied","Data":"187565bf9bb2bc0cd1971158da4f6e2fde9078ec5b1b1870f89d5b2babdd894f"} Nov 03 02:26:26.306952 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:26.306803 5463 scope.go:122] "RemoveContainer" containerID="ec5271a67eb644d44b7dde24a129d279130d039d0d2b7b35674916ae3abc4ec1" Nov 03 02:26:26.308167 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:26.308128 5463 generic.go:356] "Generic (PLEG): container finished" podID="2b8b53d3-cac3-4296-a612-e4f4604ce041" containerID="31b46eb25f4d08c2f31864aedc135b09b8e04ea16f0732dfe287dbbcf94d02d6" exitCode=0 Nov 03 02:26:26.308286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:26.308160 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq" event={"ID":"2b8b53d3-cac3-4296-a612-e4f4604ce041","Type":"ContainerDied","Data":"31b46eb25f4d08c2f31864aedc135b09b8e04ea16f0732dfe287dbbcf94d02d6"} Nov 03 02:26:26.308286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:26.308178 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq" Nov 03 02:26:26.308286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:26.308226 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq" event={"ID":"2b8b53d3-cac3-4296-a612-e4f4604ce041","Type":"ContainerDied","Data":"4ea416eb4f069b2b7920cfa5cfe61f73c86803c676b1be4d878b87abc158d850"} Nov 03 02:26:26.318888 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:26.318874 5463 scope.go:122] "RemoveContainer" containerID="ec5271a67eb644d44b7dde24a129d279130d039d0d2b7b35674916ae3abc4ec1" Nov 03 02:26:26.319155 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:26:26.319135 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec5271a67eb644d44b7dde24a129d279130d039d0d2b7b35674916ae3abc4ec1\": container with ID starting with ec5271a67eb644d44b7dde24a129d279130d039d0d2b7b35674916ae3abc4ec1 not found: ID does not exist" containerID="ec5271a67eb644d44b7dde24a129d279130d039d0d2b7b35674916ae3abc4ec1" Nov 03 02:26:26.319224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:26.319166 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5271a67eb644d44b7dde24a129d279130d039d0d2b7b35674916ae3abc4ec1"} err="failed to get container status \"ec5271a67eb644d44b7dde24a129d279130d039d0d2b7b35674916ae3abc4ec1\": rpc error: code = NotFound desc = could not find container \"ec5271a67eb644d44b7dde24a129d279130d039d0d2b7b35674916ae3abc4ec1\": container with ID starting with ec5271a67eb644d44b7dde24a129d279130d039d0d2b7b35674916ae3abc4ec1 not found: ID does not exist" Nov 03 02:26:26.319224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:26.319186 5463 scope.go:122] "RemoveContainer" containerID="31b46eb25f4d08c2f31864aedc135b09b8e04ea16f0732dfe287dbbcf94d02d6" Nov 03 02:26:26.329483 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:26.329464 5463 scope.go:122] "RemoveContainer" containerID="31b46eb25f4d08c2f31864aedc135b09b8e04ea16f0732dfe287dbbcf94d02d6" Nov 03 02:26:26.329726 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:26:26.329707 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b46eb25f4d08c2f31864aedc135b09b8e04ea16f0732dfe287dbbcf94d02d6\": container with ID starting with 31b46eb25f4d08c2f31864aedc135b09b8e04ea16f0732dfe287dbbcf94d02d6 not found: ID does not exist" containerID="31b46eb25f4d08c2f31864aedc135b09b8e04ea16f0732dfe287dbbcf94d02d6" Nov 03 02:26:26.329797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:26.329735 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b46eb25f4d08c2f31864aedc135b09b8e04ea16f0732dfe287dbbcf94d02d6"} err="failed to get container status \"31b46eb25f4d08c2f31864aedc135b09b8e04ea16f0732dfe287dbbcf94d02d6\": rpc error: code = NotFound desc = could not find container \"31b46eb25f4d08c2f31864aedc135b09b8e04ea16f0732dfe287dbbcf94d02d6\": container with ID starting with 31b46eb25f4d08c2f31864aedc135b09b8e04ea16f0732dfe287dbbcf94d02d6 not found: ID does not exist" Nov 03 02:26:26.473167 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:26.473133 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq"] Nov 03 02:26:26.521657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:26.521627 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8857cb-6cb595548f-htnxq"] Nov 03 02:26:26.582810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:26.582733 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc"] Nov 03 02:26:26.585391 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:26.585369 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-549b974859-d6fmc"] Nov 03 02:26:27.269545 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:27.269476 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:27.852860 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:27.852825 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:28.341272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:28.341236 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv"] Nov 03 02:26:28.341583 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:28.341528 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" podUID="943f92d8-5f35-4532-a30e-41961407b68b" containerName="istio-proxy" containerID="cri-o://e12b42cbc4c20c3a5fd45d787cdb52a706388901c94d3b90e92a1556ca40df0a" gracePeriod=30 Nov 03 02:26:28.360343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:28.360308 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:28.492737 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:28.492697 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b8b53d3-cac3-4296-a612-e4f4604ce041" path="/var/lib/kubelet/pods/2b8b53d3-cac3-4296-a612-e4f4604ce041/volumes" Nov 03 02:26:28.496211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:28.493336 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7c9cf2a-a41a-45c3-897e-140344ea67ff" path="/var/lib/kubelet/pods/b7c9cf2a-a41a-45c3-897e-140344ea67ff/volumes" Nov 03 02:26:29.940710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:29.940674 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:30.457380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.457351 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4"] Nov 03 02:26:30.458079 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.458057 5463 cpu_manager.go:414] "containerMap: removing container" podUID="b7c9cf2a-a41a-45c3-897e-140344ea67ff" containerName="payload-pre-processing" Nov 03 02:26:30.458079 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.458078 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c9cf2a-a41a-45c3-897e-140344ea67ff" containerName="payload-pre-processing" Nov 03 02:26:30.458301 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.458094 5463 cpu_manager.go:414] "containerMap: removing container" podUID="871baea4-93af-43d2-8ef7-0b05c4f50526" containerName="istio-proxy" Nov 03 02:26:30.458301 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.458102 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="871baea4-93af-43d2-8ef7-0b05c4f50526" containerName="istio-proxy" Nov 03 02:26:30.458301 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.458140 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2b8b53d3-cac3-4296-a612-e4f4604ce041" containerName="payload-processing" Nov 03 02:26:30.458301 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.458150 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b8b53d3-cac3-4296-a612-e4f4604ce041" containerName="payload-processing" Nov 03 02:26:30.458519 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.458320 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c9cf2a-a41a-45c3-897e-140344ea67ff" containerName="payload-pre-processing" Nov 03 02:26:30.458519 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.458338 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="871baea4-93af-43d2-8ef7-0b05c4f50526" containerName="istio-proxy" Nov 03 02:26:30.458519 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.458352 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b8b53d3-cac3-4296-a612-e4f4604ce041" containerName="payload-processing" Nov 03 02:26:30.463095 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.463074 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.561948 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.561901 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4"] Nov 03 02:26:30.563663 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.563626 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-workload-certs\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.563825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.563708 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-credential-socket\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.563825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.563762 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qk4g\" (UniqueName: \"kubernetes.io/projected/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-kube-api-access-6qk4g\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.563825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.563807 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istiod-ca-cert\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.564001 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.563848 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-workload-socket\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.564001 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.563915 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-data\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.564001 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.563942 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-token\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.564166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.564025 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-podinfo\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.564166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.564094 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-envoy\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.564335 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.564313 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:30.608658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.608625 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-7f760c-openshift-default-dockercfg-2wb7f\"" Nov 03 02:26:30.665338 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.665286 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-credential-socket\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.665536 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.665379 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qk4g\" (UniqueName: \"kubernetes.io/projected/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-kube-api-access-6qk4g\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.665536 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.665429 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istiod-ca-cert\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.665536 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.665482 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-workload-socket\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.665693 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.665553 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-data\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.665693 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.665581 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-token\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.665693 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.665649 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-podinfo\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.665865 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.665706 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-envoy\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.665865 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.665730 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-credential-socket\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.665865 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.665812 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-workload-certs\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.666039 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.665880 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-data\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.666039 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.665983 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-workload-socket\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.666152 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.666129 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-workload-certs\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.666343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.666321 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istiod-ca-cert\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.668439 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.668415 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-envoy\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.668537 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.668441 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-podinfo\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.724937 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.724873 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-token\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.725055 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.724943 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qk4g\" (UniqueName: \"kubernetes.io/projected/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-kube-api-access-6qk4g\") pod \"e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.777817 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.777776 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:30.865297 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:30.860403 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:31.209219 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:26:31.209174 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df6bd25_71cc_4cd0_bd8e_2e475b759dba.slice/crio-5dad18d8acdef5be5bd37e9ebc356f50cadffc0db23c8f9f0215178af450f427 WatchSource:0}: Error finding container 5dad18d8acdef5be5bd37e9ebc356f50cadffc0db23c8f9f0215178af450f427: Status 404 returned error can't find the container with id 5dad18d8acdef5be5bd37e9ebc356f50cadffc0db23c8f9f0215178af450f427 Nov 03 02:26:31.211779 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:31.211740 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:26:31.211895 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:31.211827 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:26:31.211895 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:31.211871 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:26:31.212901 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:31.212880 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4"] Nov 03 02:26:31.363317 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:31.363223 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" event={"ID":"9df6bd25-71cc-4cd0-bd8e-2e475b759dba","Type":"ContainerStarted","Data":"c09469c52ef697cdbb7fb2539340cd21a4d200175d5af78a7ab5bfdf2f5132d2"} Nov 03 02:26:31.363317 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:31.363292 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" event={"ID":"9df6bd25-71cc-4cd0-bd8e-2e475b759dba","Type":"ContainerStarted","Data":"5dad18d8acdef5be5bd37e9ebc356f50cadffc0db23c8f9f0215178af450f427"} Nov 03 02:26:31.481000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:31.480950 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" podStartSLOduration=1.4809360360000001 podStartE2EDuration="1.480936036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:26:31.478966757 +0000 UTC m=+2073.163075911" watchObservedRunningTime="2026-11-03 02:26:31.480936036 +0000 UTC m=+2073.165045196" Nov 03 02:26:31.778966 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:31.778932 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:32.438421 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:32.438384 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:32.784850 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:32.784769 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:33.376889 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.376860 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:33.377754 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.377732 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:26:33.623312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.623285 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:26:33.706569 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.706543 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/943f92d8-5f35-4532-a30e-41961407b68b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/943f92d8-5f35-4532-a30e-41961407b68b-istio-podinfo\") pod \"943f92d8-5f35-4532-a30e-41961407b68b\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " Nov 03 02:26:33.706729 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.706609 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-workload-certs\") pod \"943f92d8-5f35-4532-a30e-41961407b68b\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " Nov 03 02:26:33.706729 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.706661 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/943f92d8-5f35-4532-a30e-41961407b68b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/943f92d8-5f35-4532-a30e-41961407b68b-istiod-ca-cert\") pod \"943f92d8-5f35-4532-a30e-41961407b68b\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " Nov 03 02:26:33.706729 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.706704 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-workload-socket\") pod \"943f92d8-5f35-4532-a30e-41961407b68b\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " Nov 03 02:26:33.706860 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.706737 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-credential-socket\") pod \"943f92d8-5f35-4532-a30e-41961407b68b\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " Nov 03 02:26:33.706860 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.706804 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/943f92d8-5f35-4532-a30e-41961407b68b-istio-token\" (UniqueName: \"kubernetes.io/projected/943f92d8-5f35-4532-a30e-41961407b68b-istio-token\") pod \"943f92d8-5f35-4532-a30e-41961407b68b\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " Nov 03 02:26:33.706860 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.706851 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/943f92d8-5f35-4532-a30e-41961407b68b-kube-api-access-h6rc6\" (UniqueName: \"kubernetes.io/projected/943f92d8-5f35-4532-a30e-41961407b68b-kube-api-access-h6rc6\") pod \"943f92d8-5f35-4532-a30e-41961407b68b\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " Nov 03 02:26:33.707012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.706912 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-istio-envoy\") pod \"943f92d8-5f35-4532-a30e-41961407b68b\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " Nov 03 02:26:33.707012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.706978 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-istio-data\") pod \"943f92d8-5f35-4532-a30e-41961407b68b\" (UID: \"943f92d8-5f35-4532-a30e-41961407b68b\") " Nov 03 02:26:33.707165 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.707145 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-credential-socket" pod "943f92d8-5f35-4532-a30e-41961407b68b" (UID: "943f92d8-5f35-4532-a30e-41961407b68b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:26:33.707296 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.707059 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/943f92d8-5f35-4532-a30e-41961407b68b-istiod-ca-cert" pod "943f92d8-5f35-4532-a30e-41961407b68b" (UID: "943f92d8-5f35-4532-a30e-41961407b68b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:26:33.707422 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.707394 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-workload-socket" pod "943f92d8-5f35-4532-a30e-41961407b68b" (UID: "943f92d8-5f35-4532-a30e-41961407b68b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:26:33.707422 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.707407 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-istio-data" pod "943f92d8-5f35-4532-a30e-41961407b68b" (UID: "943f92d8-5f35-4532-a30e-41961407b68b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:26:33.707548 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.707419 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-workload-certs" pod "943f92d8-5f35-4532-a30e-41961407b68b" (UID: "943f92d8-5f35-4532-a30e-41961407b68b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:26:33.707619 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.707599 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:33.707680 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.707626 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:33.707680 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.707642 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/943f92d8-5f35-4532-a30e-41961407b68b-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:33.707680 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.707657 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:33.707680 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.707672 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:33.709065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.709040 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/943f92d8-5f35-4532-a30e-41961407b68b-istio-podinfo" pod "943f92d8-5f35-4532-a30e-41961407b68b" (UID: "943f92d8-5f35-4532-a30e-41961407b68b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:26:33.709317 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.709289 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/943f92d8-5f35-4532-a30e-41961407b68b-kube-api-access-h6rc6" pod "943f92d8-5f35-4532-a30e-41961407b68b" (UID: "943f92d8-5f35-4532-a30e-41961407b68b"). InnerVolumeSpecName "kube-api-access-h6rc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:26:33.709634 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.709606 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-istio-envoy" pod "943f92d8-5f35-4532-a30e-41961407b68b" (UID: "943f92d8-5f35-4532-a30e-41961407b68b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:26:33.709727 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.709634 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/943f92d8-5f35-4532-a30e-41961407b68b-istio-token" pod "943f92d8-5f35-4532-a30e-41961407b68b" (UID: "943f92d8-5f35-4532-a30e-41961407b68b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:26:33.808374 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.808340 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/943f92d8-5f35-4532-a30e-41961407b68b-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:33.808374 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.808367 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6rc6\" (UniqueName: \"kubernetes.io/projected/943f92d8-5f35-4532-a30e-41961407b68b-kube-api-access-h6rc6\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:33.808374 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.808377 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/943f92d8-5f35-4532-a30e-41961407b68b-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:33.808604 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:33.808386 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/943f92d8-5f35-4532-a30e-41961407b68b-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:34.364750 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:34.364714 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:34.384768 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:34.384740 5463 generic.go:356] "Generic (PLEG): container finished" podID="943f92d8-5f35-4532-a30e-41961407b68b" containerID="e12b42cbc4c20c3a5fd45d787cdb52a706388901c94d3b90e92a1556ca40df0a" exitCode=0 Nov 03 02:26:34.384875 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:34.384788 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" Nov 03 02:26:34.384875 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:34.384821 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" event={"ID":"943f92d8-5f35-4532-a30e-41961407b68b","Type":"ContainerDied","Data":"e12b42cbc4c20c3a5fd45d787cdb52a706388901c94d3b90e92a1556ca40df0a"} Nov 03 02:26:34.384875 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:34.384869 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv" event={"ID":"943f92d8-5f35-4532-a30e-41961407b68b","Type":"ContainerDied","Data":"6b34aafbecb6a8f183c23f6f867687e883c83fbba85517327808beb1a72af596"} Nov 03 02:26:34.385028 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:34.384892 5463 scope.go:122] "RemoveContainer" containerID="e12b42cbc4c20c3a5fd45d787cdb52a706388901c94d3b90e92a1556ca40df0a" Nov 03 02:26:34.396117 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:34.396099 5463 scope.go:122] "RemoveContainer" containerID="7b628e4470c5c1a7882801a7696f5384c05a11ac808d4fe7a016b03b729a03a7" Nov 03 02:26:34.405402 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:34.405385 5463 scope.go:122] "RemoveContainer" containerID="e12b42cbc4c20c3a5fd45d787cdb52a706388901c94d3b90e92a1556ca40df0a" Nov 03 02:26:34.405637 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:26:34.405621 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e12b42cbc4c20c3a5fd45d787cdb52a706388901c94d3b90e92a1556ca40df0a\": container with ID starting with e12b42cbc4c20c3a5fd45d787cdb52a706388901c94d3b90e92a1556ca40df0a not found: ID does not exist" containerID="e12b42cbc4c20c3a5fd45d787cdb52a706388901c94d3b90e92a1556ca40df0a" Nov 03 02:26:34.405686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:34.405643 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e12b42cbc4c20c3a5fd45d787cdb52a706388901c94d3b90e92a1556ca40df0a"} err="failed to get container status \"e12b42cbc4c20c3a5fd45d787cdb52a706388901c94d3b90e92a1556ca40df0a\": rpc error: code = NotFound desc = could not find container \"e12b42cbc4c20c3a5fd45d787cdb52a706388901c94d3b90e92a1556ca40df0a\": container with ID starting with e12b42cbc4c20c3a5fd45d787cdb52a706388901c94d3b90e92a1556ca40df0a not found: ID does not exist" Nov 03 02:26:34.405686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:34.405659 5463 scope.go:122] "RemoveContainer" containerID="7b628e4470c5c1a7882801a7696f5384c05a11ac808d4fe7a016b03b729a03a7" Nov 03 02:26:34.405873 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:26:34.405854 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b628e4470c5c1a7882801a7696f5384c05a11ac808d4fe7a016b03b729a03a7\": container with ID starting with 7b628e4470c5c1a7882801a7696f5384c05a11ac808d4fe7a016b03b729a03a7 not found: ID does not exist" containerID="7b628e4470c5c1a7882801a7696f5384c05a11ac808d4fe7a016b03b729a03a7" Nov 03 02:26:34.405916 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:34.405880 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b628e4470c5c1a7882801a7696f5384c05a11ac808d4fe7a016b03b729a03a7"} err="failed to get container status \"7b628e4470c5c1a7882801a7696f5384c05a11ac808d4fe7a016b03b729a03a7\": rpc error: code = NotFound desc = could not find container \"7b628e4470c5c1a7882801a7696f5384c05a11ac808d4fe7a016b03b729a03a7\": container with ID starting with 7b628e4470c5c1a7882801a7696f5384c05a11ac808d4fe7a016b03b729a03a7 not found: ID does not exist" Nov 03 02:26:34.541695 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:34.541645 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:34.551589 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:34.551561 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv"] Nov 03 02:26:34.601915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:34.601880 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9bd3ab-openshift-default-787778dff5-5dgqv"] Nov 03 02:26:34.709586 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:34.709561 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:34.762252 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:34.762229 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:34.780292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:34.780271 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:35.410035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:35.410006 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:26:36.442782 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:36.442746 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="943f92d8-5f35-4532-a30e-41961407b68b" path="/var/lib/kubelet/pods/943f92d8-5f35-4532-a30e-41961407b68b/volumes" Nov 03 02:26:37.302969 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.302934 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t"] Nov 03 02:26:37.303516 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.303503 5463 cpu_manager.go:414] "containerMap: removing container" podUID="943f92d8-5f35-4532-a30e-41961407b68b" containerName="istio-proxy" Nov 03 02:26:37.303568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.303518 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943f92d8-5f35-4532-a30e-41961407b68b" containerName="istio-proxy" Nov 03 02:26:37.303636 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.303627 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="943f92d8-5f35-4532-a30e-41961407b68b" containerName="istio-proxy" Nov 03 02:26:37.308756 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.308735 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" Nov 03 02:26:37.358539 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.358503 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t"] Nov 03 02:26:37.405923 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.405894 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-f760d3-serving-cert\"" Nov 03 02:26:37.448943 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.448911 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d814d4b-c428-47d5-a594-578ba0b6bfd9-metrics-certs\") pod \"maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t\" (UID: \"5d814d4b-c428-47d5-a594-578ba0b6bfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" Nov 03 02:26:37.449327 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.448955 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d814d4b-c428-47d5-a594-578ba0b6bfd9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t\" (UID: \"5d814d4b-c428-47d5-a594-578ba0b6bfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" Nov 03 02:26:37.449327 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.449105 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d814d4b-c428-47d5-a594-578ba0b6bfd9-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t\" (UID: \"5d814d4b-c428-47d5-a594-578ba0b6bfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" Nov 03 02:26:37.449327 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.449151 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrlql\" (UniqueName: \"kubernetes.io/projected/5d814d4b-c428-47d5-a594-578ba0b6bfd9-kube-api-access-wrlql\") pod \"maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t\" (UID: \"5d814d4b-c428-47d5-a594-578ba0b6bfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" Nov 03 02:26:37.502908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.502876 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:37.550379 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.550342 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d814d4b-c428-47d5-a594-578ba0b6bfd9-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t\" (UID: \"5d814d4b-c428-47d5-a594-578ba0b6bfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" Nov 03 02:26:37.550577 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.550384 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrlql\" (UniqueName: \"kubernetes.io/projected/5d814d4b-c428-47d5-a594-578ba0b6bfd9-kube-api-access-wrlql\") pod \"maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t\" (UID: \"5d814d4b-c428-47d5-a594-578ba0b6bfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" Nov 03 02:26:37.550577 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.550522 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d814d4b-c428-47d5-a594-578ba0b6bfd9-metrics-certs\") pod \"maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t\" (UID: \"5d814d4b-c428-47d5-a594-578ba0b6bfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" Nov 03 02:26:37.550577 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.550545 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d814d4b-c428-47d5-a594-578ba0b6bfd9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t\" (UID: \"5d814d4b-c428-47d5-a594-578ba0b6bfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" Nov 03 02:26:37.551051 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.551024 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d814d4b-c428-47d5-a594-578ba0b6bfd9-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t\" (UID: \"5d814d4b-c428-47d5-a594-578ba0b6bfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" Nov 03 02:26:37.552949 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.552915 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d814d4b-c428-47d5-a594-578ba0b6bfd9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t\" (UID: \"5d814d4b-c428-47d5-a594-578ba0b6bfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" Nov 03 02:26:37.553085 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.552981 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d814d4b-c428-47d5-a594-578ba0b6bfd9-metrics-certs\") pod \"maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t\" (UID: \"5d814d4b-c428-47d5-a594-578ba0b6bfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" Nov 03 02:26:37.609625 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.609593 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrlql\" (UniqueName: \"kubernetes.io/projected/5d814d4b-c428-47d5-a594-578ba0b6bfd9-kube-api-access-wrlql\") pod \"maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t\" (UID: \"5d814d4b-c428-47d5-a594-578ba0b6bfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" Nov 03 02:26:37.622283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.622258 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" Nov 03 02:26:37.647751 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.647712 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:37.804740 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.804637 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:37.929970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.929941 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv"] Nov 03 02:26:37.930611 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.930595 5463 cpu_manager.go:414] "containerMap: removing container" podUID="943f92d8-5f35-4532-a30e-41961407b68b" containerName="istio-proxy" Nov 03 02:26:37.930681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.930612 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943f92d8-5f35-4532-a30e-41961407b68b" containerName="istio-proxy" Nov 03 02:26:37.930759 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.930750 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="943f92d8-5f35-4532-a30e-41961407b68b" containerName="istio-proxy" Nov 03 02:26:37.935505 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.935487 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv" Nov 03 02:26:37.986230 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.986167 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv"] Nov 03 02:26:37.988145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:37.988121 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-7f760c\"" Nov 03 02:26:38.044222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.043479 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-7f760c-dockercfg-np6gb\"" Nov 03 02:26:38.064258 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.064161 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdcw9\" (UniqueName: \"kubernetes.io/projected/49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135-kube-api-access-vdcw9\") pod \"payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv\" (UID: \"49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv" Nov 03 02:26:38.064513 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.064492 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv\" (UID: \"49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv" Nov 03 02:26:38.087737 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.087517 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t"] Nov 03 02:26:38.096900 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.096785 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh"] Nov 03 02:26:38.101613 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.101589 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh" Nov 03 02:26:38.138216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.138178 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh"] Nov 03 02:26:38.165706 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.165675 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e7a3f30-dace-4684-9257-b2b103a2af17-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh\" (UID: \"2e7a3f30-dace-4684-9257-b2b103a2af17\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh" Nov 03 02:26:38.165866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.165743 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdcw9\" (UniqueName: \"kubernetes.io/projected/49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135-kube-api-access-vdcw9\") pod \"payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv\" (UID: \"49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv" Nov 03 02:26:38.165917 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.165890 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv\" (UID: \"49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv" Nov 03 02:26:38.166013 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.165985 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trdrx\" (UniqueName: \"kubernetes.io/projected/2e7a3f30-dace-4684-9257-b2b103a2af17-kube-api-access-trdrx\") pod \"payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh\" (UID: \"2e7a3f30-dace-4684-9257-b2b103a2af17\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh" Nov 03 02:26:38.166836 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.166813 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv\" (UID: \"49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv" Nov 03 02:26:38.222589 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.222553 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdcw9\" (UniqueName: \"kubernetes.io/projected/49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135-kube-api-access-vdcw9\") pod \"payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv\" (UID: \"49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv" Nov 03 02:26:38.253240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.253210 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv" Nov 03 02:26:38.266812 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.266791 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trdrx\" (UniqueName: \"kubernetes.io/projected/2e7a3f30-dace-4684-9257-b2b103a2af17-kube-api-access-trdrx\") pod \"payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh\" (UID: \"2e7a3f30-dace-4684-9257-b2b103a2af17\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh" Nov 03 02:26:38.266925 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.266910 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e7a3f30-dace-4684-9257-b2b103a2af17-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh\" (UID: \"2e7a3f30-dace-4684-9257-b2b103a2af17\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh" Nov 03 02:26:38.267686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.267656 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e7a3f30-dace-4684-9257-b2b103a2af17-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh\" (UID: \"2e7a3f30-dace-4684-9257-b2b103a2af17\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh" Nov 03 02:26:38.325708 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.325676 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trdrx\" (UniqueName: \"kubernetes.io/projected/2e7a3f30-dace-4684-9257-b2b103a2af17-kube-api-access-trdrx\") pod \"payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh\" (UID: \"2e7a3f30-dace-4684-9257-b2b103a2af17\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh" Nov 03 02:26:38.416363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.416336 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh" Nov 03 02:26:38.421010 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.420980 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" event={"ID":"5d814d4b-c428-47d5-a594-578ba0b6bfd9","Type":"ContainerStarted","Data":"ec6b811f78618a2ae825f3dc0c55fcfed3e3cd0362d8c0fa07cc9eff7b8df7c4"} Nov 03 02:26:38.623736 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:26:38.623705 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a6408e_d9c9_4cf3_9c6f_80a3f2a6e135.slice/crio-2958554582e6f8482fe39034b4a18a52274651c6ecb76e007485419f3623ddfe WatchSource:0}: Error finding container 2958554582e6f8482fe39034b4a18a52274651c6ecb76e007485419f3623ddfe: Status 404 returned error can't find the container with id 2958554582e6f8482fe39034b4a18a52274651c6ecb76e007485419f3623ddfe Nov 03 02:26:38.626854 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.626826 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv"] Nov 03 02:26:38.786781 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:26:38.786746 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7a3f30_dace_4684_9257_b2b103a2af17.slice/crio-079b63befe50e2c39f58629e98589abc296fe91e19846be61e7656a7480a954e WatchSource:0}: Error finding container 079b63befe50e2c39f58629e98589abc296fe91e19846be61e7656a7480a954e: Status 404 returned error can't find the container with id 079b63befe50e2c39f58629e98589abc296fe91e19846be61e7656a7480a954e Nov 03 02:26:38.789986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:38.789961 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh"] Nov 03 02:26:39.430597 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:39.430561 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" event={"ID":"5d814d4b-c428-47d5-a594-578ba0b6bfd9","Type":"ContainerStarted","Data":"39f8272e37db55f612249805bbc7facfe3463dcae6fac1fb5d749400b278af0a"} Nov 03 02:26:39.430788 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:39.430632 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" Nov 03 02:26:39.432350 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:39.432323 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv" event={"ID":"49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135","Type":"ContainerStarted","Data":"b1eaff6411e933bc3c157cdc6c483e5dc7d68babd83e694f8b62cc305296bbfd"} Nov 03 02:26:39.432500 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:39.432372 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv" event={"ID":"49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135","Type":"ContainerStarted","Data":"2958554582e6f8482fe39034b4a18a52274651c6ecb76e007485419f3623ddfe"} Nov 03 02:26:39.432500 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:39.432400 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv" Nov 03 02:26:39.433981 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:39.433960 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh" event={"ID":"2e7a3f30-dace-4684-9257-b2b103a2af17","Type":"ContainerStarted","Data":"0b7ebdd6bdf2b63c2ba3fde0911e5c69dbb0536cd3e96d384b17a275690b0e95"} Nov 03 02:26:39.434083 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:39.434008 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh" event={"ID":"2e7a3f30-dace-4684-9257-b2b103a2af17","Type":"ContainerStarted","Data":"079b63befe50e2c39f58629e98589abc296fe91e19846be61e7656a7480a954e"} Nov 03 02:26:39.434083 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:39.434039 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh" Nov 03 02:26:39.549678 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:39.549628 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" podStartSLOduration=1.6074883679999998 podStartE2EDuration="2.549615596s" totalImagesPullingTime="942.127228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:26:39.548705807 +0000 UTC m=+2081.232814967" watchObservedRunningTime="2026-11-03 02:26:39.549615596 +0000 UTC m=+2081.233724773" Nov 03 02:26:39.662442 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:39.662390 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh" podStartSLOduration=1.662373751 podStartE2EDuration="1.662373751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:26:39.661643179 +0000 UTC m=+2081.345752336" watchObservedRunningTime="2026-11-03 02:26:39.662373751 +0000 UTC m=+2081.346482905" Nov 03 02:26:39.776733 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:39.776580 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv" podStartSLOduration=2.77656326 podStartE2EDuration="2.77656326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:26:39.775904761 +0000 UTC m=+2081.460013930" watchObservedRunningTime="2026-11-03 02:26:39.77656326 +0000 UTC m=+2081.460672415" Nov 03 02:26:42.717658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:42.717617 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:43.007648 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:26:43.007569 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:26:45.448238 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:45.448207 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" Nov 03 02:26:47.419108 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:47.419070 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:50.445100 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:50.445070 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh" Nov 03 02:26:50.445635 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:50.445139 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv" Nov 03 02:26:52.004958 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:52.004919 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:52.179665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:52.179628 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:52.335911 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:52.335831 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:56.192649 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.192610 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd"] Nov 03 02:26:56.193137 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.192857 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" podUID="de3acde6-6c41-4773-a7e0-f4a7b32a6d66" containerName="maas-api" containerID="cri-o://2391079a1580d9aa082895151664202fc36330cbb35a25246c5a54c44fc8b634" gracePeriod=30 Nov 03 02:26:56.310092 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.310055 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:56.487407 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.484924 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" Nov 03 02:26:56.532640 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.532615 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-openshift-service-ca\") pod \"de3acde6-6c41-4773-a7e0-f4a7b32a6d66\" (UID: \"de3acde6-6c41-4773-a7e0-f4a7b32a6d66\") " Nov 03 02:26:56.532775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.532760 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-metrics-certs\") pod \"de3acde6-6c41-4773-a7e0-f4a7b32a6d66\" (UID: \"de3acde6-6c41-4773-a7e0-f4a7b32a6d66\") " Nov 03 02:26:56.532820 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.532793 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-maas-api-tls\") pod \"de3acde6-6c41-4773-a7e0-f4a7b32a6d66\" (UID: \"de3acde6-6c41-4773-a7e0-f4a7b32a6d66\") " Nov 03 02:26:56.532894 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.532876 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-kube-api-access-fxlrv\" (UniqueName: \"kubernetes.io/projected/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-kube-api-access-fxlrv\") pod \"de3acde6-6c41-4773-a7e0-f4a7b32a6d66\" (UID: \"de3acde6-6c41-4773-a7e0-f4a7b32a6d66\") " Nov 03 02:26:56.532932 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.532905 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-openshift-service-ca" pod "de3acde6-6c41-4773-a7e0-f4a7b32a6d66" (UID: "de3acde6-6c41-4773-a7e0-f4a7b32a6d66"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:26:56.533380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.533361 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:56.535001 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.534980 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-metrics-certs" pod "de3acde6-6c41-4773-a7e0-f4a7b32a6d66" (UID: "de3acde6-6c41-4773-a7e0-f4a7b32a6d66"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:26:56.535521 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.535497 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-maas-api-tls" pod "de3acde6-6c41-4773-a7e0-f4a7b32a6d66" (UID: "de3acde6-6c41-4773-a7e0-f4a7b32a6d66"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:26:56.535601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.535502 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-kube-api-access-fxlrv" pod "de3acde6-6c41-4773-a7e0-f4a7b32a6d66" (UID: "de3acde6-6c41-4773-a7e0-f4a7b32a6d66"). InnerVolumeSpecName "kube-api-access-fxlrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:26:56.607207 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.602221 5463 generic.go:356] "Generic (PLEG): container finished" podID="de3acde6-6c41-4773-a7e0-f4a7b32a6d66" containerID="2391079a1580d9aa082895151664202fc36330cbb35a25246c5a54c44fc8b634" exitCode=0 Nov 03 02:26:56.607207 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.602568 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" event={"ID":"de3acde6-6c41-4773-a7e0-f4a7b32a6d66","Type":"ContainerDied","Data":"2391079a1580d9aa082895151664202fc36330cbb35a25246c5a54c44fc8b634"} Nov 03 02:26:56.607207 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.602649 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" event={"ID":"de3acde6-6c41-4773-a7e0-f4a7b32a6d66","Type":"ContainerDied","Data":"4a4c94c2364df25e42625e9a5c32ae0eb679b6029d49fbadeede9dcfb7f48263"} Nov 03 02:26:56.607207 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.602709 5463 scope.go:122] "RemoveContainer" containerID="2391079a1580d9aa082895151664202fc36330cbb35a25246c5a54c44fc8b634" Nov 03 02:26:56.607207 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.602940 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd" Nov 03 02:26:56.623408 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.623385 5463 scope.go:122] "RemoveContainer" containerID="2391079a1580d9aa082895151664202fc36330cbb35a25246c5a54c44fc8b634" Nov 03 02:26:56.623735 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:26:56.623707 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2391079a1580d9aa082895151664202fc36330cbb35a25246c5a54c44fc8b634\": container with ID starting with 2391079a1580d9aa082895151664202fc36330cbb35a25246c5a54c44fc8b634 not found: ID does not exist" containerID="2391079a1580d9aa082895151664202fc36330cbb35a25246c5a54c44fc8b634" Nov 03 02:26:56.623817 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.623748 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2391079a1580d9aa082895151664202fc36330cbb35a25246c5a54c44fc8b634"} err="failed to get container status \"2391079a1580d9aa082895151664202fc36330cbb35a25246c5a54c44fc8b634\": rpc error: code = NotFound desc = could not find container \"2391079a1580d9aa082895151664202fc36330cbb35a25246c5a54c44fc8b634\": container with ID starting with 2391079a1580d9aa082895151664202fc36330cbb35a25246c5a54c44fc8b634 not found: ID does not exist" Nov 03 02:26:56.634878 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.634852 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:56.634878 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.634875 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:56.635003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.634889 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxlrv\" (UniqueName: \"kubernetes.io/projected/de3acde6-6c41-4773-a7e0-f4a7b32a6d66-kube-api-access-fxlrv\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:56.773161 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.773067 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd"] Nov 03 02:26:56.776764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:56.776731 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d63a25-57b446fbbd-vvzcd"] Nov 03 02:26:58.179034 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:58.178997 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:58.443967 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:58.443893 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de3acde6-6c41-4773-a7e0-f4a7b32a6d66" path="/var/lib/kubelet/pods/de3acde6-6c41-4773-a7e0-f4a7b32a6d66/volumes" Nov 03 02:26:59.626610 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:59.626526 5463 generic.go:356] "Generic (PLEG): container finished" podID="d029bf9f-23a3-47c7-8eca-e343baed7d99" containerID="93e1f9bbae83197112663116fd74171229e18aef93fe1ea898fbd10f5b7296fb" exitCode=0 Nov 03 02:26:59.626610 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:59.626596 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" event={"ID":"d029bf9f-23a3-47c7-8eca-e343baed7d99","Type":"ContainerDied","Data":"93e1f9bbae83197112663116fd74171229e18aef93fe1ea898fbd10f5b7296fb"} Nov 03 02:26:59.627163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:59.627012 5463 scope.go:122] "RemoveContainer" containerID="93e1f9bbae83197112663116fd74171229e18aef93fe1ea898fbd10f5b7296fb" Nov 03 02:26:59.628602 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:59.628570 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:26:59.628713 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:59.628638 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:26:59.628713 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:26:59.628667 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:27:00.635260 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:00.635225 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" event={"ID":"d029bf9f-23a3-47c7-8eca-e343baed7d99","Type":"ContainerStarted","Data":"99cfafc85940344d8c897518cbc3a4442ffc8f6ab01716f2e1f713ab98543008"} Nov 03 02:27:00.666215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:00.661586 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:27:00.666215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:00.661712 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 03 02:27:00.666215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:00.661773 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" podUID="d029bf9f-23a3-47c7-8eca-e343baed7d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 03 02:27:01.334861 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:01.334820 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:01.485787 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:01.485749 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:01.657538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:01.657501 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 03 02:27:01.658128 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:01.658096 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" podUID="d029bf9f-23a3-47c7-8eca-e343baed7d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 03 02:27:02.666215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:02.662550 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 03 02:27:02.666215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:02.662620 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" podUID="d029bf9f-23a3-47c7-8eca-e343baed7d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 03 02:27:02.707783 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:02.707754 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:27:02.713298 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:02.713275 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:27:02.904039 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:02.904008 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:03.387095 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:03.387055 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:03.390577 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:03.390553 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22"] Nov 03 02:27:03.390813 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:03.390789 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" podUID="b2a15ca1-f5c6-41af-849f-9e727badc8a7" containerName="istio-proxy" containerID="cri-o://8ce30abaaf6a34be9771565169b2a6619dba3633f41309fdbe31766297ff0168" gracePeriod=30 Nov 03 02:27:03.666330 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:03.666295 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 03 02:27:03.666915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:03.666887 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" podUID="d029bf9f-23a3-47c7-8eca-e343baed7d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 03 02:27:04.368042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:04.368007 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:04.673216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:04.672590 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 03 02:27:04.673216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:04.672671 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" podUID="d029bf9f-23a3-47c7-8eca-e343baed7d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 03 02:27:04.942635 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:04.942552 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:05.168570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.168533 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj"] Nov 03 02:27:05.169790 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.169755 5463 cpu_manager.go:414] "containerMap: removing container" podUID="de3acde6-6c41-4773-a7e0-f4a7b32a6d66" containerName="maas-api" Nov 03 02:27:05.169947 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.169932 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de3acde6-6c41-4773-a7e0-f4a7b32a6d66" containerName="maas-api" Nov 03 02:27:05.170381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.170363 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="de3acde6-6c41-4773-a7e0-f4a7b32a6d66" containerName="maas-api" Nov 03 02:27:05.177014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.176994 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.229641 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.229561 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-fb1940-openshift-default-dockercfg-ks8b6\"" Nov 03 02:27:05.243141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.243114 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45226bbf-4673-4108-bb2d-962a40340fb6-istio-podinfo\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.243288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.243174 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45226bbf-4673-4108-bb2d-962a40340fb6-istio-token\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.243288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.243277 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-workload-socket\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.243390 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.243331 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-istio-data\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.243390 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.243383 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45226bbf-4673-4108-bb2d-962a40340fb6-istiod-ca-cert\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.243497 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.243440 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-credential-socket\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.243497 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.243459 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-workload-certs\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.243597 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.243575 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-istio-envoy\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.243640 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.243631 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p298s\" (UniqueName: \"kubernetes.io/projected/45226bbf-4673-4108-bb2d-962a40340fb6-kube-api-access-p298s\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.285245 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.274386 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:05.291917 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.288702 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj"] Nov 03 02:27:05.345173 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.345142 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45226bbf-4673-4108-bb2d-962a40340fb6-istiod-ca-cert\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.345393 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.345233 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-credential-socket\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.345393 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.345254 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-workload-certs\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.345393 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.345292 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-istio-envoy\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.345562 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.345502 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p298s\" (UniqueName: \"kubernetes.io/projected/45226bbf-4673-4108-bb2d-962a40340fb6-kube-api-access-p298s\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.345635 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.345585 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45226bbf-4673-4108-bb2d-962a40340fb6-istio-podinfo\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.345749 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.345727 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-workload-certs\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.345822 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.345755 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-credential-socket\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.345906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.345886 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45226bbf-4673-4108-bb2d-962a40340fb6-istiod-ca-cert\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.346127 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.346103 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45226bbf-4673-4108-bb2d-962a40340fb6-istio-token\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.346238 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.346156 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-workload-socket\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.346238 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.346225 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-istio-data\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.346502 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.346482 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-workload-socket\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.346563 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.346540 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-istio-data\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.356212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.352768 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45226bbf-4673-4108-bb2d-962a40340fb6-istio-podinfo\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.379600 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.379569 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-istio-envoy\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.402735 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.402695 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45226bbf-4673-4108-bb2d-962a40340fb6-istio-token\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.402883 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.402781 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p298s\" (UniqueName: \"kubernetes.io/projected/45226bbf-4673-4108-bb2d-962a40340fb6-kube-api-access-p298s\") pod \"e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.490505 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.490427 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:05.557214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.554083 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:05.658634 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.658599 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 03 02:27:05.658809 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.658655 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" podUID="d029bf9f-23a3-47c7-8eca-e343baed7d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 03 02:27:05.964282 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:27:05.964253 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45226bbf_4673_4108_bb2d_962a40340fb6.slice/crio-39dc40e89441ad3476fef9ff5fca7466f1b6cc2bd353a1a02d0b3428eb3f07f1 WatchSource:0}: Error finding container 39dc40e89441ad3476fef9ff5fca7466f1b6cc2bd353a1a02d0b3428eb3f07f1: Status 404 returned error can't find the container with id 39dc40e89441ad3476fef9ff5fca7466f1b6cc2bd353a1a02d0b3428eb3f07f1 Nov 03 02:27:05.967055 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.967032 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj"] Nov 03 02:27:05.967161 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.967133 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:27:05.967225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.967215 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:27:05.967268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:05.967246 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:27:06.662799 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:06.662770 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:27:06.663072 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:06.663047 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:27:06.663902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:06.663878 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:27:06.703632 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:06.703604 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" event={"ID":"45226bbf-4673-4108-bb2d-962a40340fb6","Type":"ContainerStarted","Data":"37642d83a1157566381133b6f350635afe59ea09ffc227a8193d3cb5f95acf75"} Nov 03 02:27:06.703797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:06.703647 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" event={"ID":"45226bbf-4673-4108-bb2d-962a40340fb6","Type":"ContainerStarted","Data":"39dc40e89441ad3476fef9ff5fca7466f1b6cc2bd353a1a02d0b3428eb3f07f1"} Nov 03 02:27:06.895933 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:06.895883 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" podStartSLOduration=1.895869391 podStartE2EDuration="1.895869391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:27:06.894507291 +0000 UTC m=+2108.578616457" watchObservedRunningTime="2026-11-03 02:27:06.895869391 +0000 UTC m=+2108.579978547" Nov 03 02:27:07.490992 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:07.490954 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:07.497433 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:07.497396 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:07.711580 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:07.711546 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:07.712368 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:07.712346 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:27:08.664887 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.664834 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:27:08.693154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.693125 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-workload-certs\") pod \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " Nov 03 02:27:08.693303 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.693167 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-podinfo\") pod \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " Nov 03 02:27:08.693303 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.693275 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-data\") pod \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " Nov 03 02:27:08.693419 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.693306 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-envoy\") pod \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " Nov 03 02:27:08.693419 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.693361 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-token\" (UniqueName: \"kubernetes.io/projected/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-token\") pod \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " Nov 03 02:27:08.693419 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.693404 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-workload-certs" pod "b2a15ca1-f5c6-41af-849f-9e727badc8a7" (UID: "b2a15ca1-f5c6-41af-849f-9e727badc8a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:27:08.693584 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.693458 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-credential-socket\") pod \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " Nov 03 02:27:08.693584 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.693508 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a15ca1-f5c6-41af-849f-9e727badc8a7-kube-api-access-fjcnd\" (UniqueName: \"kubernetes.io/projected/b2a15ca1-f5c6-41af-849f-9e727badc8a7-kube-api-access-fjcnd\") pod \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " Nov 03 02:27:08.693690 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.693584 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istiod-ca-cert\") pod \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " Nov 03 02:27:08.693690 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.693588 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-data" pod "b2a15ca1-f5c6-41af-849f-9e727badc8a7" (UID: "b2a15ca1-f5c6-41af-849f-9e727badc8a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:27:08.693830 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.693804 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-credential-socket" pod "b2a15ca1-f5c6-41af-849f-9e727badc8a7" (UID: "b2a15ca1-f5c6-41af-849f-9e727badc8a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:27:08.694163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.694143 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istiod-ca-cert" pod "b2a15ca1-f5c6-41af-849f-9e727badc8a7" (UID: "b2a15ca1-f5c6-41af-849f-9e727badc8a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:27:08.694298 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.694261 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:27:08.694298 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.694285 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:27:08.694381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.694303 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:27:08.695933 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.695904 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-token" pod "b2a15ca1-f5c6-41af-849f-9e727badc8a7" (UID: "b2a15ca1-f5c6-41af-849f-9e727badc8a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:27:08.696059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.696038 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a15ca1-f5c6-41af-849f-9e727badc8a7-kube-api-access-fjcnd" pod "b2a15ca1-f5c6-41af-849f-9e727badc8a7" (UID: "b2a15ca1-f5c6-41af-849f-9e727badc8a7"). InnerVolumeSpecName "kube-api-access-fjcnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:27:08.696306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.696281 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-podinfo" pod "b2a15ca1-f5c6-41af-849f-9e727badc8a7" (UID: "b2a15ca1-f5c6-41af-849f-9e727badc8a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:27:08.697320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.697295 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-envoy" pod "b2a15ca1-f5c6-41af-849f-9e727badc8a7" (UID: "b2a15ca1-f5c6-41af-849f-9e727badc8a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:27:08.719480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.719451 5463 generic.go:356] "Generic (PLEG): container finished" podID="b2a15ca1-f5c6-41af-849f-9e727badc8a7" containerID="8ce30abaaf6a34be9771565169b2a6619dba3633f41309fdbe31766297ff0168" exitCode=0 Nov 03 02:27:08.719596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.719500 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" Nov 03 02:27:08.719596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.719526 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" event={"ID":"b2a15ca1-f5c6-41af-849f-9e727badc8a7","Type":"ContainerDied","Data":"8ce30abaaf6a34be9771565169b2a6619dba3633f41309fdbe31766297ff0168"} Nov 03 02:27:08.719596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.719577 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22" event={"ID":"b2a15ca1-f5c6-41af-849f-9e727badc8a7","Type":"ContainerDied","Data":"9465165317a02d1661bd188ec60013b84d18dcdc858b6379d91c96dec86014cc"} Nov 03 02:27:08.719761 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.719599 5463 scope.go:122] "RemoveContainer" containerID="8ce30abaaf6a34be9771565169b2a6619dba3633f41309fdbe31766297ff0168" Nov 03 02:27:08.729550 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.729532 5463 scope.go:122] "RemoveContainer" containerID="53792e0b9d91a6c631fdd5b4462e467a020b9048d53f72f129dd7dd57e8ab792" Nov 03 02:27:08.738620 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.738597 5463 scope.go:122] "RemoveContainer" containerID="8ce30abaaf6a34be9771565169b2a6619dba3633f41309fdbe31766297ff0168" Nov 03 02:27:08.738922 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:27:08.738903 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce30abaaf6a34be9771565169b2a6619dba3633f41309fdbe31766297ff0168\": container with ID starting with 8ce30abaaf6a34be9771565169b2a6619dba3633f41309fdbe31766297ff0168 not found: ID does not exist" containerID="8ce30abaaf6a34be9771565169b2a6619dba3633f41309fdbe31766297ff0168" Nov 03 02:27:08.739000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.738931 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce30abaaf6a34be9771565169b2a6619dba3633f41309fdbe31766297ff0168"} err="failed to get container status \"8ce30abaaf6a34be9771565169b2a6619dba3633f41309fdbe31766297ff0168\": rpc error: code = NotFound desc = could not find container \"8ce30abaaf6a34be9771565169b2a6619dba3633f41309fdbe31766297ff0168\": container with ID starting with 8ce30abaaf6a34be9771565169b2a6619dba3633f41309fdbe31766297ff0168 not found: ID does not exist" Nov 03 02:27:08.739000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.738949 5463 scope.go:122] "RemoveContainer" containerID="53792e0b9d91a6c631fdd5b4462e467a020b9048d53f72f129dd7dd57e8ab792" Nov 03 02:27:08.739240 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:27:08.739222 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53792e0b9d91a6c631fdd5b4462e467a020b9048d53f72f129dd7dd57e8ab792\": container with ID starting with 53792e0b9d91a6c631fdd5b4462e467a020b9048d53f72f129dd7dd57e8ab792 not found: ID does not exist" containerID="53792e0b9d91a6c631fdd5b4462e467a020b9048d53f72f129dd7dd57e8ab792" Nov 03 02:27:08.739319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.739250 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53792e0b9d91a6c631fdd5b4462e467a020b9048d53f72f129dd7dd57e8ab792"} err="failed to get container status \"53792e0b9d91a6c631fdd5b4462e467a020b9048d53f72f129dd7dd57e8ab792\": rpc error: code = NotFound desc = could not find container \"53792e0b9d91a6c631fdd5b4462e467a020b9048d53f72f129dd7dd57e8ab792\": container with ID starting with 53792e0b9d91a6c631fdd5b4462e467a020b9048d53f72f129dd7dd57e8ab792 not found: ID does not exist" Nov 03 02:27:08.794775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.794746 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-workload-socket\") pod \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\" (UID: \"b2a15ca1-f5c6-41af-849f-9e727badc8a7\") " Nov 03 02:27:08.795034 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.794998 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-workload-socket" pod "b2a15ca1-f5c6-41af-849f-9e727badc8a7" (UID: "b2a15ca1-f5c6-41af-849f-9e727badc8a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:27:08.795577 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.795424 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjcnd\" (UniqueName: \"kubernetes.io/projected/b2a15ca1-f5c6-41af-849f-9e727badc8a7-kube-api-access-fjcnd\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:27:08.795577 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.795450 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:27:08.795577 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.795465 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:27:08.795577 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.795480 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:27:08.795577 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.795494 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:27:08.795577 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:08.795508 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a15ca1-f5c6-41af-849f-9e727badc8a7-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:27:09.142570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:09.142537 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22"] Nov 03 02:27:09.145502 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:09.145460 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d63a25-openshift-default-6c6965d4d9-xbm22"] Nov 03 02:27:09.700757 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:09.700722 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:09.876798 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:09.876740 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:10.033005 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:10.032928 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:10.202618 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:10.202585 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:10.443110 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:10.443077 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2a15ca1-f5c6-41af-849f-9e727badc8a7" path="/var/lib/kubelet/pods/b2a15ca1-f5c6-41af-849f-9e727badc8a7/volumes" Nov 03 02:27:12.972668 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:12.972625 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:13.018914 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:27:13.018841 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:27:13.118094 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.118063 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:13.311764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.311690 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq"] Nov 03 02:27:13.312294 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.312279 5463 cpu_manager.go:414] "containerMap: removing container" podUID="b2a15ca1-f5c6-41af-849f-9e727badc8a7" containerName="istio-proxy" Nov 03 02:27:13.312348 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.312295 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a15ca1-f5c6-41af-849f-9e727badc8a7" containerName="istio-proxy" Nov 03 02:27:13.312424 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.312415 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a15ca1-f5c6-41af-849f-9e727badc8a7" containerName="istio-proxy" Nov 03 02:27:13.312460 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.312428 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a15ca1-f5c6-41af-849f-9e727badc8a7" containerName="istio-proxy" Nov 03 02:27:13.315832 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.315816 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq" Nov 03 02:27:13.370301 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.370274 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq"] Nov 03 02:27:13.416013 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.415987 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fb1940\"" Nov 03 02:27:13.484370 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.484334 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t6tk\" (UniqueName: \"kubernetes.io/projected/3d7216d2-aafe-4501-9085-38ee727a1b97-kube-api-access-8t6tk\") pod \"payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq\" (UID: \"3d7216d2-aafe-4501-9085-38ee727a1b97\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq" Nov 03 02:27:13.484575 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.484394 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d7216d2-aafe-4501-9085-38ee727a1b97-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq\" (UID: \"3d7216d2-aafe-4501-9085-38ee727a1b97\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq" Nov 03 02:27:13.488325 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.487732 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824"] Nov 03 02:27:13.489471 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.489447 5463 cpu_manager.go:414] "containerMap: removing container" podUID="b2a15ca1-f5c6-41af-849f-9e727badc8a7" containerName="istio-proxy" Nov 03 02:27:13.489612 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.489599 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a15ca1-f5c6-41af-849f-9e727badc8a7" containerName="istio-proxy" Nov 03 02:27:13.493247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.493229 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824" Nov 03 02:27:13.528041 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.528008 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824"] Nov 03 02:27:13.546331 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.546301 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fb1940-dockercfg-5m5h9\"" Nov 03 02:27:13.586301 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.586138 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839fcc2f-d52f-48fe-be5c-918dd68526f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824\" (UID: \"839fcc2f-d52f-48fe-be5c-918dd68526f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824" Nov 03 02:27:13.586301 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.586184 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p7tv\" (UniqueName: \"kubernetes.io/projected/839fcc2f-d52f-48fe-be5c-918dd68526f4-kube-api-access-2p7tv\") pod \"payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824\" (UID: \"839fcc2f-d52f-48fe-be5c-918dd68526f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824" Nov 03 02:27:13.586524 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.586491 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t6tk\" (UniqueName: \"kubernetes.io/projected/3d7216d2-aafe-4501-9085-38ee727a1b97-kube-api-access-8t6tk\") pod \"payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq\" (UID: \"3d7216d2-aafe-4501-9085-38ee727a1b97\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq" Nov 03 02:27:13.586589 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.586558 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d7216d2-aafe-4501-9085-38ee727a1b97-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq\" (UID: \"3d7216d2-aafe-4501-9085-38ee727a1b97\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq" Nov 03 02:27:13.587711 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.587689 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d7216d2-aafe-4501-9085-38ee727a1b97-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq\" (UID: \"3d7216d2-aafe-4501-9085-38ee727a1b97\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq" Nov 03 02:27:13.643836 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.643810 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t6tk\" (UniqueName: \"kubernetes.io/projected/3d7216d2-aafe-4501-9085-38ee727a1b97-kube-api-access-8t6tk\") pod \"payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq\" (UID: \"3d7216d2-aafe-4501-9085-38ee727a1b97\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq" Nov 03 02:27:13.688204 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.688153 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839fcc2f-d52f-48fe-be5c-918dd68526f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824\" (UID: \"839fcc2f-d52f-48fe-be5c-918dd68526f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824" Nov 03 02:27:13.688381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.688213 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p7tv\" (UniqueName: \"kubernetes.io/projected/839fcc2f-d52f-48fe-be5c-918dd68526f4-kube-api-access-2p7tv\") pod \"payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824\" (UID: \"839fcc2f-d52f-48fe-be5c-918dd68526f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824" Nov 03 02:27:13.689012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.688987 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839fcc2f-d52f-48fe-be5c-918dd68526f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824\" (UID: \"839fcc2f-d52f-48fe-be5c-918dd68526f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824" Nov 03 02:27:13.745102 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.745075 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p7tv\" (UniqueName: \"kubernetes.io/projected/839fcc2f-d52f-48fe-be5c-918dd68526f4-kube-api-access-2p7tv\") pod \"payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824\" (UID: \"839fcc2f-d52f-48fe-be5c-918dd68526f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824" Nov 03 02:27:13.813186 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.813157 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824" Nov 03 02:27:13.930126 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:13.930095 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq" Nov 03 02:27:14.094050 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:14.094019 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:14.184000 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:27:14.183970 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839fcc2f_d52f_48fe_be5c_918dd68526f4.slice/crio-bc2f3306f422bc50e8dae95bb94cd7002849135fdd20f0d5c606821df2a15420 WatchSource:0}: Error finding container bc2f3306f422bc50e8dae95bb94cd7002849135fdd20f0d5c606821df2a15420: Status 404 returned error can't find the container with id bc2f3306f422bc50e8dae95bb94cd7002849135fdd20f0d5c606821df2a15420 Nov 03 02:27:14.187415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:14.187393 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824"] Nov 03 02:27:14.318595 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:14.313554 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq"] Nov 03 02:27:14.575157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:14.575064 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:14.796213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:14.795248 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq" event={"ID":"3d7216d2-aafe-4501-9085-38ee727a1b97","Type":"ContainerStarted","Data":"a6da390b481538b088d0507401371d97e5dbe3d0fcef24b0542bf71e6cecb636"} Nov 03 02:27:14.796213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:14.795361 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq" event={"ID":"3d7216d2-aafe-4501-9085-38ee727a1b97","Type":"ContainerStarted","Data":"b1b893d268d7b6aa64a15e12562e5c2d1cf4f892aa8f6c64d656d16f2792c14d"} Nov 03 02:27:14.799171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:14.796481 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq" Nov 03 02:27:14.799171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:14.798588 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824" event={"ID":"839fcc2f-d52f-48fe-be5c-918dd68526f4","Type":"ContainerStarted","Data":"4ffa72d3b14e65b755835dc77fbd2872d4c9b4da43f8f27cdaab5b3054e80849"} Nov 03 02:27:14.799171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:14.798645 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824" event={"ID":"839fcc2f-d52f-48fe-be5c-918dd68526f4","Type":"ContainerStarted","Data":"bc2f3306f422bc50e8dae95bb94cd7002849135fdd20f0d5c606821df2a15420"} Nov 03 02:27:14.799565 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:14.799547 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824" Nov 03 02:27:14.913717 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:14.913660 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq" podStartSLOduration=1.913641182 podStartE2EDuration="1.913641182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:27:14.909755727 +0000 UTC m=+2116.593864882" watchObservedRunningTime="2026-11-03 02:27:14.913641182 +0000 UTC m=+2116.597750339" Nov 03 02:27:14.941021 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:14.940992 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:15.657844 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:15.657786 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824" podStartSLOduration=2.657771465 podStartE2EDuration="2.657771465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:27:15.029039677 +0000 UTC m=+2116.713148832" watchObservedRunningTime="2026-11-03 02:27:15.657771465 +0000 UTC m=+2117.341880623" Nov 03 02:27:15.659424 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:15.659402 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq"] Nov 03 02:27:15.662635 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:15.662614 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq" Nov 03 02:27:15.715534 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:15.715500 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq"] Nov 03 02:27:15.762296 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:15.762260 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8857cb\"" Nov 03 02:27:15.762296 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:15.762260 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8857cb-dockercfg-jtgsd\"" Nov 03 02:27:15.817640 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:15.817602 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smz6l\" (UniqueName: \"kubernetes.io/projected/dad5987b-deb7-4e6a-b8a5-8e2821700969-kube-api-access-smz6l\") pod \"payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq\" (UID: \"dad5987b-deb7-4e6a-b8a5-8e2821700969\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq" Nov 03 02:27:15.817902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:15.817818 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad5987b-deb7-4e6a-b8a5-8e2821700969-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq\" (UID: \"dad5987b-deb7-4e6a-b8a5-8e2821700969\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq" Nov 03 02:27:15.919213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:15.919111 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smz6l\" (UniqueName: \"kubernetes.io/projected/dad5987b-deb7-4e6a-b8a5-8e2821700969-kube-api-access-smz6l\") pod \"payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq\" (UID: \"dad5987b-deb7-4e6a-b8a5-8e2821700969\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq" Nov 03 02:27:15.919370 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:15.919322 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad5987b-deb7-4e6a-b8a5-8e2821700969-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq\" (UID: \"dad5987b-deb7-4e6a-b8a5-8e2821700969\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq" Nov 03 02:27:15.920084 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:15.920061 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad5987b-deb7-4e6a-b8a5-8e2821700969-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq\" (UID: \"dad5987b-deb7-4e6a-b8a5-8e2821700969\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq" Nov 03 02:27:15.976021 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:15.975968 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smz6l\" (UniqueName: \"kubernetes.io/projected/dad5987b-deb7-4e6a-b8a5-8e2821700969-kube-api-access-smz6l\") pod \"payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq\" (UID: \"dad5987b-deb7-4e6a-b8a5-8e2821700969\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq" Nov 03 02:27:15.977731 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:15.977708 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq" Nov 03 02:27:16.346324 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:27:16.346288 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad5987b_deb7_4e6a_b8a5_8e2821700969.slice/crio-b710546e8e342765c3d338c7e26a8dcd77af736a27f6a811b0811ffa0187e221 WatchSource:0}: Error finding container b710546e8e342765c3d338c7e26a8dcd77af736a27f6a811b0811ffa0187e221: Status 404 returned error can't find the container with id b710546e8e342765c3d338c7e26a8dcd77af736a27f6a811b0811ffa0187e221 Nov 03 02:27:16.349828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:16.349802 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq"] Nov 03 02:27:16.458366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:16.458333 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s"] Nov 03 02:27:16.462566 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:16.462545 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s" Nov 03 02:27:16.512096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:16.512066 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s"] Nov 03 02:27:16.627304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:16.627228 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6xd6\" (UniqueName: \"kubernetes.io/projected/2063d869-2397-4105-bdb3-ea049d4bd196-kube-api-access-q6xd6\") pod \"payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s\" (UID: \"2063d869-2397-4105-bdb3-ea049d4bd196\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s" Nov 03 02:27:16.627304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:16.627267 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2063d869-2397-4105-bdb3-ea049d4bd196-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s\" (UID: \"2063d869-2397-4105-bdb3-ea049d4bd196\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s" Nov 03 02:27:16.728507 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:16.728476 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6xd6\" (UniqueName: \"kubernetes.io/projected/2063d869-2397-4105-bdb3-ea049d4bd196-kube-api-access-q6xd6\") pod \"payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s\" (UID: \"2063d869-2397-4105-bdb3-ea049d4bd196\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s" Nov 03 02:27:16.728883 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:16.728518 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2063d869-2397-4105-bdb3-ea049d4bd196-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s\" (UID: \"2063d869-2397-4105-bdb3-ea049d4bd196\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s" Nov 03 02:27:16.729275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:16.729245 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2063d869-2397-4105-bdb3-ea049d4bd196-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s\" (UID: \"2063d869-2397-4105-bdb3-ea049d4bd196\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s" Nov 03 02:27:16.786429 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:16.786394 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6xd6\" (UniqueName: \"kubernetes.io/projected/2063d869-2397-4105-bdb3-ea049d4bd196-kube-api-access-q6xd6\") pod \"payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s\" (UID: \"2063d869-2397-4105-bdb3-ea049d4bd196\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s" Nov 03 02:27:16.822016 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:16.821967 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq" event={"ID":"dad5987b-deb7-4e6a-b8a5-8e2821700969","Type":"ContainerStarted","Data":"953dff0687d13f8a9d209bbd05c5a3c2b46cddbf3fb7aa46e618463c9db63f9c"} Nov 03 02:27:16.822177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:16.822039 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq" event={"ID":"dad5987b-deb7-4e6a-b8a5-8e2821700969","Type":"ContainerStarted","Data":"b710546e8e342765c3d338c7e26a8dcd77af736a27f6a811b0811ffa0187e221"} Nov 03 02:27:16.822177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:16.822073 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq" Nov 03 02:27:16.940170 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:16.940124 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq" podStartSLOduration=1.940111221 podStartE2EDuration="1.940111221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:27:16.93727121 +0000 UTC m=+2118.621380367" watchObservedRunningTime="2026-11-03 02:27:16.940111221 +0000 UTC m=+2118.624220376" Nov 03 02:27:17.076821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:17.076789 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s" Nov 03 02:27:17.159545 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:17.159018 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:17.447313 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:27:17.447283 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2063d869_2397_4105_bdb3_ea049d4bd196.slice/crio-d1277fdcdd3e2b1a2bf258bdf969aa73e778771e6dd0ba81e63fe8e9d5e720b9 WatchSource:0}: Error finding container d1277fdcdd3e2b1a2bf258bdf969aa73e778771e6dd0ba81e63fe8e9d5e720b9: Status 404 returned error can't find the container with id d1277fdcdd3e2b1a2bf258bdf969aa73e778771e6dd0ba81e63fe8e9d5e720b9 Nov 03 02:27:17.450508 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:17.450480 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s"] Nov 03 02:27:17.831265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:17.831162 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s" event={"ID":"2063d869-2397-4105-bdb3-ea049d4bd196","Type":"ContainerStarted","Data":"5835b707361de2750c95a27b6f3e448f169270632f5506bf49a705b58ea0ccc4"} Nov 03 02:27:17.831265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:17.831238 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s" event={"ID":"2063d869-2397-4105-bdb3-ea049d4bd196","Type":"ContainerStarted","Data":"d1277fdcdd3e2b1a2bf258bdf969aa73e778771e6dd0ba81e63fe8e9d5e720b9"} Nov 03 02:27:17.831778 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:17.831291 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s" Nov 03 02:27:17.948519 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:17.948458 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s" podStartSLOduration=1.9484394059999999 podStartE2EDuration="1.948439406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:27:17.945363517 +0000 UTC m=+2119.629472671" watchObservedRunningTime="2026-11-03 02:27:17.948439406 +0000 UTC m=+2119.632548563" Nov 03 02:27:21.826338 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:21.826291 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:26.505752 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:26.505668 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:26.825148 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:26.825071 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq" Nov 03 02:27:26.825346 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:26.825153 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824" Nov 03 02:27:27.833725 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:27.833690 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq" Nov 03 02:27:28.841183 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:28.841153 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s" Nov 03 02:27:29.965479 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:29.965447 5463 generic.go:356] "Generic (PLEG): container finished" podID="cbecd0cc-1384-46b8-8eaa-758627373292" containerID="fd2edee22c38d95c9e6e3273e986c4ecd2c743c1efeb7c94a413853922eb347c" exitCode=0 Nov 03 02:27:29.965945 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:29.965516 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" event={"ID":"cbecd0cc-1384-46b8-8eaa-758627373292","Type":"ContainerDied","Data":"fd2edee22c38d95c9e6e3273e986c4ecd2c743c1efeb7c94a413853922eb347c"} Nov 03 02:27:29.965945 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:29.965895 5463 scope.go:122] "RemoveContainer" containerID="fd2edee22c38d95c9e6e3273e986c4ecd2c743c1efeb7c94a413853922eb347c" Nov 03 02:27:29.967397 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:29.967368 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:27:29.967471 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:29.967444 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:27:29.967512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:29.967479 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:27:30.984151 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:30.984108 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" event={"ID":"cbecd0cc-1384-46b8-8eaa-758627373292","Type":"ContainerStarted","Data":"c7a1a56fbf01f2ed0f3e06f56d0b5a699859e7a3a02a25b8c8c6bcc6006ba227"} Nov 03 02:27:31.226177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:31.226139 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:31.290710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:31.290611 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:27:31.292562 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:31.292530 5463 patch_prober.go:29] interesting pod/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 03 02:27:31.292722 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:31.292594 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" podUID="cbecd0cc-1384-46b8-8eaa-758627373292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 03 02:27:32.283822 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:32.283789 5463 patch_prober.go:29] interesting pod/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 03 02:27:32.284294 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:32.283844 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" podUID="cbecd0cc-1384-46b8-8eaa-758627373292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 03 02:27:33.283785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:33.283747 5463 patch_prober.go:29] interesting pod/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 03 02:27:33.283975 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:33.283807 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" podUID="cbecd0cc-1384-46b8-8eaa-758627373292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 03 02:27:34.283387 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:34.283351 5463 patch_prober.go:29] interesting pod/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 03 02:27:34.283573 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:34.283424 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" podUID="cbecd0cc-1384-46b8-8eaa-758627373292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 03 02:27:34.980444 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:34.980412 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:27:35.283543 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:35.283462 5463 patch_prober.go:29] interesting pod/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 03 02:27:35.283543 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:35.283516 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" podUID="cbecd0cc-1384-46b8-8eaa-758627373292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 03 02:27:35.870509 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:35.870446 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:36.032474 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:36.032434 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:36.181186 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:36.181154 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:36.287861 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:36.287834 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:27:36.289038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:36.289016 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:27:40.736618 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:40.736583 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:43.008398 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:27:43.008363 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:27:45.525480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:45.525438 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:46.883539 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:46.883504 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:48.243580 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:48.243471 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:48.942512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:48.942473 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:49.159584 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:49.159554 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:50.633397 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:50.633364 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:53.164216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:53.161832 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:53.337629 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:53.337553 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:53.488435 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:53.488315 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:55.710644 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:55.710607 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:56.362431 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:56.362388 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:56.538902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:56.538872 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:56.757642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:56.757597 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg"] Nov 03 02:27:56.762609 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:56.762584 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg" Nov 03 02:27:56.814661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:56.814119 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg"] Nov 03 02:27:56.881595 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:56.863018 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-496177\"" Nov 03 02:27:56.881595 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:56.863180 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-496177-dockercfg-mdtxg\"" Nov 03 02:27:56.916146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:56.916105 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnr4f\" (UniqueName: \"kubernetes.io/projected/29bbc3e3-3107-45c7-bd50-ee262c18c015-kube-api-access-vnr4f\") pod \"payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg\" (UID: \"29bbc3e3-3107-45c7-bd50-ee262c18c015\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg" Nov 03 02:27:56.916510 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:56.916490 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29bbc3e3-3107-45c7-bd50-ee262c18c015-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg\" (UID: \"29bbc3e3-3107-45c7-bd50-ee262c18c015\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg" Nov 03 02:27:57.018258 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:57.018146 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29bbc3e3-3107-45c7-bd50-ee262c18c015-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg\" (UID: \"29bbc3e3-3107-45c7-bd50-ee262c18c015\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg" Nov 03 02:27:57.018450 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:57.018430 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnr4f\" (UniqueName: \"kubernetes.io/projected/29bbc3e3-3107-45c7-bd50-ee262c18c015-kube-api-access-vnr4f\") pod \"payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg\" (UID: \"29bbc3e3-3107-45c7-bd50-ee262c18c015\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg" Nov 03 02:27:57.018968 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:57.018950 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29bbc3e3-3107-45c7-bd50-ee262c18c015-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg\" (UID: \"29bbc3e3-3107-45c7-bd50-ee262c18c015\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg" Nov 03 02:27:57.075877 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:57.075837 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnr4f\" (UniqueName: \"kubernetes.io/projected/29bbc3e3-3107-45c7-bd50-ee262c18c015-kube-api-access-vnr4f\") pod \"payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg\" (UID: \"29bbc3e3-3107-45c7-bd50-ee262c18c015\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg" Nov 03 02:27:57.082026 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:57.081992 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg" Nov 03 02:27:57.503745 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:57.503716 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg"] Nov 03 02:27:58.223386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:58.223345 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg" event={"ID":"29bbc3e3-3107-45c7-bd50-ee262c18c015","Type":"ContainerStarted","Data":"3c37d6bf12602f5f80d8a9504f1a05ae362b91e6f8327c66add47598988465e0"} Nov 03 02:27:58.223890 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:58.223441 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg" event={"ID":"29bbc3e3-3107-45c7-bd50-ee262c18c015","Type":"ContainerStarted","Data":"b066bd3dafba8d222ae83479126f2f97bd87d82d6c01f32b31b2b9fcdb30eff1"} Nov 03 02:27:58.223890 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:58.223473 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg" Nov 03 02:27:58.338475 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:58.338419 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg" podStartSLOduration=2.338404889 podStartE2EDuration="2.338404889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:27:58.336068938 +0000 UTC m=+2160.020178094" watchObservedRunningTime="2026-11-03 02:27:58.338404889 +0000 UTC m=+2160.022514044" Nov 03 02:27:59.233411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:59.233332 5463 generic.go:356] "Generic (PLEG): container finished" podID="9df6bd25-71cc-4cd0-bd8e-2e475b759dba" containerID="c09469c52ef697cdbb7fb2539340cd21a4d200175d5af78a7ab5bfdf2f5132d2" exitCode=0 Nov 03 02:27:59.233850 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:59.233410 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" event={"ID":"9df6bd25-71cc-4cd0-bd8e-2e475b759dba","Type":"ContainerDied","Data":"c09469c52ef697cdbb7fb2539340cd21a4d200175d5af78a7ab5bfdf2f5132d2"} Nov 03 02:27:59.233968 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:59.233944 5463 scope.go:122] "RemoveContainer" containerID="c09469c52ef697cdbb7fb2539340cd21a4d200175d5af78a7ab5bfdf2f5132d2" Nov 03 02:27:59.235738 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:59.235700 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:27:59.235841 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:59.235793 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:27:59.235887 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:27:59.235840 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:28:00.241874 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:00.241844 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" event={"ID":"9df6bd25-71cc-4cd0-bd8e-2e475b759dba","Type":"ContainerStarted","Data":"f84d4c0317b2616e3640920ee43a050cbf143e61a638534a0d3e770c6b1bd87a"} Nov 03 02:28:00.774216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:00.773878 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:00.782213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:00.782109 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:28:00.784166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:00.784134 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 03 02:28:00.784358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:00.784219 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" podUID="9df6bd25-71cc-4cd0-bd8e-2e475b759dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 03 02:28:01.778486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:01.778449 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 03 02:28:01.778889 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:01.778512 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" podUID="9df6bd25-71cc-4cd0-bd8e-2e475b759dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 03 02:28:02.778395 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:02.778361 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 03 02:28:02.778596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:02.778414 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" podUID="9df6bd25-71cc-4cd0-bd8e-2e475b759dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 03 02:28:03.376838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:03.376803 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:28:03.778460 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:03.778424 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 03 02:28:03.778669 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:03.778514 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" podUID="9df6bd25-71cc-4cd0-bd8e-2e475b759dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 03 02:28:04.778664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:04.778623 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 03 02:28:04.779160 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:04.778694 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" podUID="9df6bd25-71cc-4cd0-bd8e-2e475b759dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 03 02:28:04.967214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:04.962407 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:05.783570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:05.783537 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:28:05.784719 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:05.784695 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:28:07.418544 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:07.418511 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:09.237084 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:09.237004 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg" Nov 03 02:28:10.115272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:10.115239 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:11.797217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:11.797172 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m"] Nov 03 02:28:11.801359 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:11.801341 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" Nov 03 02:28:11.855454 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:11.855423 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m"] Nov 03 02:28:11.900329 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:11.900303 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-fb1940-serving-cert\"" Nov 03 02:28:11.930862 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:11.930833 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbzbq\" (UniqueName: \"kubernetes.io/projected/6d372c46-0419-4d51-a4e6-0e651b656331-kube-api-access-nbzbq\") pod \"maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m\" (UID: \"6d372c46-0419-4d51-a4e6-0e651b656331\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" Nov 03 02:28:11.931001 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:11.930887 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d372c46-0419-4d51-a4e6-0e651b656331-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m\" (UID: \"6d372c46-0419-4d51-a4e6-0e651b656331\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" Nov 03 02:28:11.931054 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:11.931042 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d372c46-0419-4d51-a4e6-0e651b656331-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m\" (UID: \"6d372c46-0419-4d51-a4e6-0e651b656331\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" Nov 03 02:28:11.931166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:11.931150 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d372c46-0419-4d51-a4e6-0e651b656331-metrics-certs\") pod \"maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m\" (UID: \"6d372c46-0419-4d51-a4e6-0e651b656331\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" Nov 03 02:28:12.000059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:12.000015 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:12.032027 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:12.031998 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbzbq\" (UniqueName: \"kubernetes.io/projected/6d372c46-0419-4d51-a4e6-0e651b656331-kube-api-access-nbzbq\") pod \"maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m\" (UID: \"6d372c46-0419-4d51-a4e6-0e651b656331\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" Nov 03 02:28:12.032180 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:12.032049 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d372c46-0419-4d51-a4e6-0e651b656331-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m\" (UID: \"6d372c46-0419-4d51-a4e6-0e651b656331\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" Nov 03 02:28:12.032180 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:12.032110 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d372c46-0419-4d51-a4e6-0e651b656331-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m\" (UID: \"6d372c46-0419-4d51-a4e6-0e651b656331\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" Nov 03 02:28:12.032180 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:12.032171 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d372c46-0419-4d51-a4e6-0e651b656331-metrics-certs\") pod \"maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m\" (UID: \"6d372c46-0419-4d51-a4e6-0e651b656331\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" Nov 03 02:28:12.032336 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:28:12.032205 5463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-serving-cert: secret "maas-api-e2e-ipp-a-fb1940-serving-cert" not found Nov 03 02:28:12.032336 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:28:12.032261 5463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d372c46-0419-4d51-a4e6-0e651b656331-maas-api-tls podName:6d372c46-0419-4d51-a4e6-0e651b656331 nodeName:}" failed. No retries permitted until 2026-11-03 02:28:12.53224645 +0000 UTC m=+2174.216355582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6d372c46-0419-4d51-a4e6-0e651b656331-maas-api-tls") pod "maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" (UID: "6d372c46-0419-4d51-a4e6-0e651b656331") : secret "maas-api-e2e-ipp-a-fb1940-serving-cert" not found Nov 03 02:28:12.033028 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:12.033007 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d372c46-0419-4d51-a4e6-0e651b656331-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m\" (UID: \"6d372c46-0419-4d51-a4e6-0e651b656331\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" Nov 03 02:28:12.037115 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:12.035473 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d372c46-0419-4d51-a4e6-0e651b656331-metrics-certs\") pod \"maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m\" (UID: \"6d372c46-0419-4d51-a4e6-0e651b656331\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" Nov 03 02:28:12.092740 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:12.092661 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbzbq\" (UniqueName: \"kubernetes.io/projected/6d372c46-0419-4d51-a4e6-0e651b656331-kube-api-access-nbzbq\") pod \"maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m\" (UID: \"6d372c46-0419-4d51-a4e6-0e651b656331\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" Nov 03 02:28:12.181273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:12.181239 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:12.330639 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:12.330611 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:12.538662 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:12.538628 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d372c46-0419-4d51-a4e6-0e651b656331-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m\" (UID: \"6d372c46-0419-4d51-a4e6-0e651b656331\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" Nov 03 02:28:12.541005 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:12.540981 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d372c46-0419-4d51-a4e6-0e651b656331-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m\" (UID: \"6d372c46-0419-4d51-a4e6-0e651b656331\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" Nov 03 02:28:12.655164 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:12.655128 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:12.715126 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:12.715098 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" Nov 03 02:28:12.802036 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:12.801937 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:13.011103 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:28:13.011064 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:28:13.092395 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:28:13.092364 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d372c46_0419_4d51_a4e6_0e651b656331.slice/crio-b077191daaea2f864026f672e33ee2330296f1123ba196ae16b656f3b85ac600 WatchSource:0}: Error finding container b077191daaea2f864026f672e33ee2330296f1123ba196ae16b656f3b85ac600: Status 404 returned error can't find the container with id b077191daaea2f864026f672e33ee2330296f1123ba196ae16b656f3b85ac600 Nov 03 02:28:13.094596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:13.094571 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m"] Nov 03 02:28:13.360856 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:13.360773 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" event={"ID":"6d372c46-0419-4d51-a4e6-0e651b656331","Type":"ContainerStarted","Data":"b077191daaea2f864026f672e33ee2330296f1123ba196ae16b656f3b85ac600"} Nov 03 02:28:14.369953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:14.369859 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" event={"ID":"6d372c46-0419-4d51-a4e6-0e651b656331","Type":"ContainerStarted","Data":"72de326e507553fb8b26d057c959a1d98bf29ae39ccbc9cd6b4b8157eded1d07"} Nov 03 02:28:14.369953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:14.369927 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" Nov 03 02:28:14.486477 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:14.486425 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" podStartSLOduration=2.603650106 podStartE2EDuration="3.486407399s" totalImagesPullingTime="882.757293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:28:14.486167355 +0000 UTC m=+2176.170276509" watchObservedRunningTime="2026-11-03 02:28:14.486407399 +0000 UTC m=+2176.170516551" Nov 03 02:28:15.260598 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:15.260558 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:15.974454 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:15.974383 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:17.125854 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:17.125821 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:18.871327 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:18.871285 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv"] Nov 03 02:28:18.978474 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:18.978442 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp"] Nov 03 02:28:19.241787 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:19.241758 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:19.392281 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:19.392241 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:20.385402 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:20.385362 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" Nov 03 02:28:21.366163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:21.366120 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:23.854549 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:23.854516 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:23.871936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:23.871877 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv" podUID="f0bb7921-0ba1-4ea6-b010-504d816c2d4a" containerName="payload-processing" containerID="cri-o://5c2c20b57e9eb4a11bfe629f0db8de0740ef61170aa2818f478bab6825fb05a5" gracePeriod=25 Nov 03 02:28:23.979235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:23.979178 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp" podUID="0c862e45-235b-4ee4-9efd-094389f6f071" containerName="payload-pre-processing" containerID="cri-o://8cc28b3d7a61a3acdaa71af608c74ad70bdee75cdd028fe378d65bd7dcab0798" gracePeriod=25 Nov 03 02:28:24.156095 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.156064 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv" Nov 03 02:28:24.202803 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.202768 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0bb7921-0ba1-4ea6-b010-504d816c2d4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0bb7921-0ba1-4ea6-b010-504d816c2d4a-plugins-config-volume\") pod \"f0bb7921-0ba1-4ea6-b010-504d816c2d4a\" (UID: \"f0bb7921-0ba1-4ea6-b010-504d816c2d4a\") " Nov 03 02:28:24.203046 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.203025 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0bb7921-0ba1-4ea6-b010-504d816c2d4a-kube-api-access-44qgs\" (UniqueName: \"kubernetes.io/projected/f0bb7921-0ba1-4ea6-b010-504d816c2d4a-kube-api-access-44qgs\") pod \"f0bb7921-0ba1-4ea6-b010-504d816c2d4a\" (UID: \"f0bb7921-0ba1-4ea6-b010-504d816c2d4a\") " Nov 03 02:28:24.203999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.203940 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bb7921-0ba1-4ea6-b010-504d816c2d4a-plugins-config-volume" pod "f0bb7921-0ba1-4ea6-b010-504d816c2d4a" (UID: "f0bb7921-0ba1-4ea6-b010-504d816c2d4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:28:24.205320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.205293 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bb7921-0ba1-4ea6-b010-504d816c2d4a-kube-api-access-44qgs" pod "f0bb7921-0ba1-4ea6-b010-504d816c2d4a" (UID: "f0bb7921-0ba1-4ea6-b010-504d816c2d4a"). InnerVolumeSpecName "kube-api-access-44qgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:28:24.258140 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.258114 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp" Nov 03 02:28:24.304170 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.304133 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c862e45-235b-4ee4-9efd-094389f6f071-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c862e45-235b-4ee4-9efd-094389f6f071-plugins-config-volume\") pod \"0c862e45-235b-4ee4-9efd-094389f6f071\" (UID: \"0c862e45-235b-4ee4-9efd-094389f6f071\") " Nov 03 02:28:24.304384 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.304363 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c862e45-235b-4ee4-9efd-094389f6f071-kube-api-access-s4dg6\" (UniqueName: \"kubernetes.io/projected/0c862e45-235b-4ee4-9efd-094389f6f071-kube-api-access-s4dg6\") pod \"0c862e45-235b-4ee4-9efd-094389f6f071\" (UID: \"0c862e45-235b-4ee4-9efd-094389f6f071\") " Nov 03 02:28:24.304548 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.304522 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c862e45-235b-4ee4-9efd-094389f6f071-plugins-config-volume" pod "0c862e45-235b-4ee4-9efd-094389f6f071" (UID: "0c862e45-235b-4ee4-9efd-094389f6f071"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:28:24.304913 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.304890 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44qgs\" (UniqueName: \"kubernetes.io/projected/f0bb7921-0ba1-4ea6-b010-504d816c2d4a-kube-api-access-44qgs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:24.304994 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.304929 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0bb7921-0ba1-4ea6-b010-504d816c2d4a-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:24.304994 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.304945 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c862e45-235b-4ee4-9efd-094389f6f071-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:24.307229 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.307205 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c862e45-235b-4ee4-9efd-094389f6f071-kube-api-access-s4dg6" pod "0c862e45-235b-4ee4-9efd-094389f6f071" (UID: "0c862e45-235b-4ee4-9efd-094389f6f071"). InnerVolumeSpecName "kube-api-access-s4dg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:28:24.405852 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.405818 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4dg6\" (UniqueName: \"kubernetes.io/projected/0c862e45-235b-4ee4-9efd-094389f6f071-kube-api-access-s4dg6\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:24.447386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.447350 5463 generic.go:356] "Generic (PLEG): container finished" podID="0c862e45-235b-4ee4-9efd-094389f6f071" containerID="8cc28b3d7a61a3acdaa71af608c74ad70bdee75cdd028fe378d65bd7dcab0798" exitCode=0 Nov 03 02:28:24.447592 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.447415 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp" Nov 03 02:28:24.447592 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.447443 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp" event={"ID":"0c862e45-235b-4ee4-9efd-094389f6f071","Type":"ContainerDied","Data":"8cc28b3d7a61a3acdaa71af608c74ad70bdee75cdd028fe378d65bd7dcab0798"} Nov 03 02:28:24.447592 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.447498 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp" event={"ID":"0c862e45-235b-4ee4-9efd-094389f6f071","Type":"ContainerDied","Data":"69d3c668139fec1df88e8a8518e1f6fac74dceedefc27ae7148c625dcc2bfd02"} Nov 03 02:28:24.447592 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.447528 5463 scope.go:122] "RemoveContainer" containerID="8cc28b3d7a61a3acdaa71af608c74ad70bdee75cdd028fe378d65bd7dcab0798" Nov 03 02:28:24.449289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.449166 5463 generic.go:356] "Generic (PLEG): container finished" podID="f0bb7921-0ba1-4ea6-b010-504d816c2d4a" containerID="5c2c20b57e9eb4a11bfe629f0db8de0740ef61170aa2818f478bab6825fb05a5" exitCode=0 Nov 03 02:28:24.449457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.449438 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv" event={"ID":"f0bb7921-0ba1-4ea6-b010-504d816c2d4a","Type":"ContainerDied","Data":"5c2c20b57e9eb4a11bfe629f0db8de0740ef61170aa2818f478bab6825fb05a5"} Nov 03 02:28:24.449529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.449484 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv" event={"ID":"f0bb7921-0ba1-4ea6-b010-504d816c2d4a","Type":"ContainerDied","Data":"7d49ab62475f9c8fefe141d5224e56453393f9995a23e128e4f94ed38f437ed5"} Nov 03 02:28:24.449595 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.449578 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv" Nov 03 02:28:24.462696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.462679 5463 scope.go:122] "RemoveContainer" containerID="8cc28b3d7a61a3acdaa71af608c74ad70bdee75cdd028fe378d65bd7dcab0798" Nov 03 02:28:24.462957 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:28:24.462936 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc28b3d7a61a3acdaa71af608c74ad70bdee75cdd028fe378d65bd7dcab0798\": container with ID starting with 8cc28b3d7a61a3acdaa71af608c74ad70bdee75cdd028fe378d65bd7dcab0798 not found: ID does not exist" containerID="8cc28b3d7a61a3acdaa71af608c74ad70bdee75cdd028fe378d65bd7dcab0798" Nov 03 02:28:24.463006 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.462965 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc28b3d7a61a3acdaa71af608c74ad70bdee75cdd028fe378d65bd7dcab0798"} err="failed to get container status \"8cc28b3d7a61a3acdaa71af608c74ad70bdee75cdd028fe378d65bd7dcab0798\": rpc error: code = NotFound desc = could not find container \"8cc28b3d7a61a3acdaa71af608c74ad70bdee75cdd028fe378d65bd7dcab0798\": container with ID starting with 8cc28b3d7a61a3acdaa71af608c74ad70bdee75cdd028fe378d65bd7dcab0798 not found: ID does not exist" Nov 03 02:28:24.463006 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.462983 5463 scope.go:122] "RemoveContainer" containerID="5c2c20b57e9eb4a11bfe629f0db8de0740ef61170aa2818f478bab6825fb05a5" Nov 03 02:28:24.472214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.472177 5463 scope.go:122] "RemoveContainer" containerID="5c2c20b57e9eb4a11bfe629f0db8de0740ef61170aa2818f478bab6825fb05a5" Nov 03 02:28:24.472473 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:28:24.472454 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2c20b57e9eb4a11bfe629f0db8de0740ef61170aa2818f478bab6825fb05a5\": container with ID starting with 5c2c20b57e9eb4a11bfe629f0db8de0740ef61170aa2818f478bab6825fb05a5 not found: ID does not exist" containerID="5c2c20b57e9eb4a11bfe629f0db8de0740ef61170aa2818f478bab6825fb05a5" Nov 03 02:28:24.472526 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.472483 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2c20b57e9eb4a11bfe629f0db8de0740ef61170aa2818f478bab6825fb05a5"} err="failed to get container status \"5c2c20b57e9eb4a11bfe629f0db8de0740ef61170aa2818f478bab6825fb05a5\": rpc error: code = NotFound desc = could not find container \"5c2c20b57e9eb4a11bfe629f0db8de0740ef61170aa2818f478bab6825fb05a5\": container with ID starting with 5c2c20b57e9eb4a11bfe629f0db8de0740ef61170aa2818f478bab6825fb05a5 not found: ID does not exist" Nov 03 02:28:24.612944 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.612912 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv"] Nov 03 02:28:24.616289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.616264 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c62c3d-79966549fc-jz6sv"] Nov 03 02:28:24.725074 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.724994 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp"] Nov 03 02:28:24.729060 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:24.729035 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c62c3d-7b557966c9-qszrp"] Nov 03 02:28:26.443967 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:26.443933 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c862e45-235b-4ee4-9efd-094389f6f071" path="/var/lib/kubelet/pods/0c862e45-235b-4ee4-9efd-094389f6f071/volumes" Nov 03 02:28:26.444551 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:26.444528 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0bb7921-0ba1-4ea6-b010-504d816c2d4a" path="/var/lib/kubelet/pods/f0bb7921-0ba1-4ea6-b010-504d816c2d4a/volumes" Nov 03 02:28:28.534442 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:28.534412 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:33.299367 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:33.299323 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:37.413151 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:37.413112 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:39.104517 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:39.104486 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:43.010796 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:28:43.010757 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:28:43.625016 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:43.624983 5463 generic.go:356] "Generic (PLEG): container finished" podID="45226bbf-4673-4108-bb2d-962a40340fb6" containerID="37642d83a1157566381133b6f350635afe59ea09ffc227a8193d3cb5f95acf75" exitCode=0 Nov 03 02:28:43.625263 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:43.625061 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" event={"ID":"45226bbf-4673-4108-bb2d-962a40340fb6","Type":"ContainerDied","Data":"37642d83a1157566381133b6f350635afe59ea09ffc227a8193d3cb5f95acf75"} Nov 03 02:28:43.625555 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:43.625536 5463 scope.go:122] "RemoveContainer" containerID="37642d83a1157566381133b6f350635afe59ea09ffc227a8193d3cb5f95acf75" Nov 03 02:28:43.627079 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:43.627052 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:28:43.627140 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:43.627123 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:28:43.627181 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:43.627154 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:28:43.779210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:43.778318 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:43.931112 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:43.931074 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:44.634696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:44.634665 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" event={"ID":"45226bbf-4673-4108-bb2d-962a40340fb6","Type":"ContainerStarted","Data":"6b423d45bd051448a81ba53a700b5c4dd74b47522a48e4b28d445da3db58cde3"} Nov 03 02:28:45.365217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:45.362797 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:45.491727 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:45.491620 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:28:45.492117 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:45.492086 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 03 02:28:45.492288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:45.492160 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" podUID="45226bbf-4673-4108-bb2d-962a40340fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 03 02:28:45.872226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:45.870538 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj"] Nov 03 02:28:45.872226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:45.870814 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" podUID="e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" containerName="istio-proxy" containerID="cri-o://5e009602cb886f697076313344b625625b187ca3d2faea1752a2be34da99a916" gracePeriod=30 Nov 03 02:28:45.873105 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:45.873047 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:46.491147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:46.491107 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 03 02:28:46.491385 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:46.491204 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" podUID="45226bbf-4673-4108-bb2d-962a40340fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 03 02:28:47.430554 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:47.430519 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:47.491655 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:47.491613 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 03 02:28:47.491857 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:47.491697 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" podUID="45226bbf-4673-4108-bb2d-962a40340fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 03 02:28:48.312269 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:48.312231 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:28:48.312451 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:48.312300 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" podUID="e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:28:48.491409 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:48.491368 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 03 02:28:48.491833 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:48.491437 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" podUID="45226bbf-4673-4108-bb2d-962a40340fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 03 02:28:48.706042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:48.705994 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:49.008534 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:49.008437 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:49.364731 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:49.364529 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:49.491708 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:49.491676 5463 patch_prober.go:29] interesting pod/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 03 02:28:49.492126 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:49.491727 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" podUID="45226bbf-4673-4108-bb2d-962a40340fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 03 02:28:49.695414 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:49.695364 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:50.497055 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:50.497022 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:28:50.497550 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:50.497532 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:28:50.498151 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:50.498133 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:28:51.554525 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.554498 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:28:51.654095 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.654030 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-workload-certs\") pod \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " Nov 03 02:28:51.654095 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.654068 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-workload-socket\") pod \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " Nov 03 02:28:51.654355 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.654124 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-token\" (UniqueName: \"kubernetes.io/projected/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-token\") pod \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " Nov 03 02:28:51.654355 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.654175 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istiod-ca-cert\") pod \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " Nov 03 02:28:51.654355 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.654241 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-kube-api-access-k5ssv\" (UniqueName: \"kubernetes.io/projected/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-kube-api-access-k5ssv\") pod \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " Nov 03 02:28:51.654355 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.654276 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-podinfo\") pod \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " Nov 03 02:28:51.654355 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.654315 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-workload-certs" pod "e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" (UID: "e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:28:51.654696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.654435 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-data\") pod \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " Nov 03 02:28:51.654696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.654499 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-envoy\") pod \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " Nov 03 02:28:51.654696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.654535 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-credential-socket\") pod \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\" (UID: \"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f\") " Nov 03 02:28:51.654696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.654477 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-workload-socket" pod "e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" (UID: "e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:28:51.654696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.654612 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istiod-ca-cert" pod "e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" (UID: "e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:28:51.655121 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.654781 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-data" pod "e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" (UID: "e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:28:51.655121 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.655080 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:51.655121 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.655099 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:51.655121 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.655108 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:51.655121 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.655116 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:51.655811 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.655752 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-credential-socket" pod "e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" (UID: "e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:28:51.656730 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.656706 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-token" pod "e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" (UID: "e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:28:51.657271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.657246 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-podinfo" pod "e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" (UID: "e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:28:51.657490 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.657461 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-kube-api-access-k5ssv" pod "e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" (UID: "e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f"). InnerVolumeSpecName "kube-api-access-k5ssv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:28:51.657691 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.657666 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-envoy" pod "e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" (UID: "e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:28:51.717609 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.717570 5463 generic.go:356] "Generic (PLEG): container finished" podID="e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" containerID="5e009602cb886f697076313344b625625b187ca3d2faea1752a2be34da99a916" exitCode=0 Nov 03 02:28:51.717805 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.717649 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" Nov 03 02:28:51.717805 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.717728 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" event={"ID":"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f","Type":"ContainerDied","Data":"5e009602cb886f697076313344b625625b187ca3d2faea1752a2be34da99a916"} Nov 03 02:28:51.717896 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.717810 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj" event={"ID":"e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f","Type":"ContainerDied","Data":"4696123f335b0a6b7977a3e573556e0dcb393718b136741a187bd674928837b0"} Nov 03 02:28:51.717896 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.717838 5463 scope.go:122] "RemoveContainer" containerID="5e009602cb886f697076313344b625625b187ca3d2faea1752a2be34da99a916" Nov 03 02:28:51.729356 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.729335 5463 scope.go:122] "RemoveContainer" containerID="ec0e3cbe1669b07275fd6906e388407cd2fac1893c9dba0801903c5c8bf8f3e0" Nov 03 02:28:51.738374 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.738356 5463 scope.go:122] "RemoveContainer" containerID="5e009602cb886f697076313344b625625b187ca3d2faea1752a2be34da99a916" Nov 03 02:28:51.738624 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:28:51.738602 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e009602cb886f697076313344b625625b187ca3d2faea1752a2be34da99a916\": container with ID starting with 5e009602cb886f697076313344b625625b187ca3d2faea1752a2be34da99a916 not found: ID does not exist" containerID="5e009602cb886f697076313344b625625b187ca3d2faea1752a2be34da99a916" Nov 03 02:28:51.738685 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.738631 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e009602cb886f697076313344b625625b187ca3d2faea1752a2be34da99a916"} err="failed to get container status \"5e009602cb886f697076313344b625625b187ca3d2faea1752a2be34da99a916\": rpc error: code = NotFound desc = could not find container \"5e009602cb886f697076313344b625625b187ca3d2faea1752a2be34da99a916\": container with ID starting with 5e009602cb886f697076313344b625625b187ca3d2faea1752a2be34da99a916 not found: ID does not exist" Nov 03 02:28:51.738685 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.738648 5463 scope.go:122] "RemoveContainer" containerID="ec0e3cbe1669b07275fd6906e388407cd2fac1893c9dba0801903c5c8bf8f3e0" Nov 03 02:28:51.738891 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:28:51.738873 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec0e3cbe1669b07275fd6906e388407cd2fac1893c9dba0801903c5c8bf8f3e0\": container with ID starting with ec0e3cbe1669b07275fd6906e388407cd2fac1893c9dba0801903c5c8bf8f3e0 not found: ID does not exist" containerID="ec0e3cbe1669b07275fd6906e388407cd2fac1893c9dba0801903c5c8bf8f3e0" Nov 03 02:28:51.738929 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.738897 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec0e3cbe1669b07275fd6906e388407cd2fac1893c9dba0801903c5c8bf8f3e0"} err="failed to get container status \"ec0e3cbe1669b07275fd6906e388407cd2fac1893c9dba0801903c5c8bf8f3e0\": rpc error: code = NotFound desc = could not find container \"ec0e3cbe1669b07275fd6906e388407cd2fac1893c9dba0801903c5c8bf8f3e0\": container with ID starting with ec0e3cbe1669b07275fd6906e388407cd2fac1893c9dba0801903c5c8bf8f3e0 not found: ID does not exist" Nov 03 02:28:51.756811 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.756782 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:51.756811 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.756806 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5ssv\" (UniqueName: \"kubernetes.io/projected/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-kube-api-access-k5ssv\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:51.757000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.756824 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:51.757000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.756838 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:51.757000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.756850 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:51.891279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.891247 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj"] Nov 03 02:28:51.897378 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.897345 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c62c3d-openshift-default-58b8c9489c-s9sdj"] Nov 03 02:28:51.960215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:51.960126 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:52.445574 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:52.445540 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" path="/var/lib/kubelet/pods/e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f/volumes" Nov 03 02:28:53.489278 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:53.489246 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:53.700357 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:53.700320 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:53.863240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:53.863129 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:54.341207 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:54.341169 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:56.882614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:56.882582 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:57.084647 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.084602 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:57.426538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.426505 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv"] Nov 03 02:28:57.427245 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.427226 5463 cpu_manager.go:414] "containerMap: removing container" podUID="e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" containerName="istio-proxy" Nov 03 02:28:57.427245 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.427246 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" containerName="istio-proxy" Nov 03 02:28:57.427373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.427260 5463 cpu_manager.go:414] "containerMap: removing container" podUID="f0bb7921-0ba1-4ea6-b010-504d816c2d4a" containerName="payload-processing" Nov 03 02:28:57.427373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.427266 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0bb7921-0ba1-4ea6-b010-504d816c2d4a" containerName="payload-processing" Nov 03 02:28:57.427373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.427291 5463 cpu_manager.go:414] "containerMap: removing container" podUID="e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" containerName="istio-proxy" Nov 03 02:28:57.427373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.427297 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" containerName="istio-proxy" Nov 03 02:28:57.427373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.427314 5463 cpu_manager.go:414] "containerMap: removing container" podUID="0c862e45-235b-4ee4-9efd-094389f6f071" containerName="payload-pre-processing" Nov 03 02:28:57.427373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.427319 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c862e45-235b-4ee4-9efd-094389f6f071" containerName="payload-pre-processing" Nov 03 02:28:57.427623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.427418 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" containerName="istio-proxy" Nov 03 02:28:57.427623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.427427 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0bb7921-0ba1-4ea6-b010-504d816c2d4a" containerName="payload-processing" Nov 03 02:28:57.427623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.427439 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c862e45-235b-4ee4-9efd-094389f6f071" containerName="payload-pre-processing" Nov 03 02:28:57.430999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.430979 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv" Nov 03 02:28:57.483517 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.483490 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-711318-dockercfg-gvpkl\"" Nov 03 02:28:57.484445 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.484425 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv"] Nov 03 02:28:57.525990 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.525954 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6421252b-76d2-40ed-af4c-9e8086e8b87b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-711318-675db99787-mk5zv\" (UID: \"6421252b-76d2-40ed-af4c-9e8086e8b87b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv" Nov 03 02:28:57.526119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.526080 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtrck\" (UniqueName: \"kubernetes.io/projected/6421252b-76d2-40ed-af4c-9e8086e8b87b-kube-api-access-wtrck\") pod \"payload-processing-e2e-praxis-b-711318-675db99787-mk5zv\" (UID: \"6421252b-76d2-40ed-af4c-9e8086e8b87b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv" Nov 03 02:28:57.531557 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.531537 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-711318\"" Nov 03 02:28:57.626993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.626965 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6421252b-76d2-40ed-af4c-9e8086e8b87b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-711318-675db99787-mk5zv\" (UID: \"6421252b-76d2-40ed-af4c-9e8086e8b87b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv" Nov 03 02:28:57.627157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.627041 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtrck\" (UniqueName: \"kubernetes.io/projected/6421252b-76d2-40ed-af4c-9e8086e8b87b-kube-api-access-wtrck\") pod \"payload-processing-e2e-praxis-b-711318-675db99787-mk5zv\" (UID: \"6421252b-76d2-40ed-af4c-9e8086e8b87b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv" Nov 03 02:28:57.627736 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.627716 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6421252b-76d2-40ed-af4c-9e8086e8b87b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-711318-675db99787-mk5zv\" (UID: \"6421252b-76d2-40ed-af4c-9e8086e8b87b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv" Nov 03 02:28:57.684182 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.684112 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtrck\" (UniqueName: \"kubernetes.io/projected/6421252b-76d2-40ed-af4c-9e8086e8b87b-kube-api-access-wtrck\") pod \"payload-processing-e2e-praxis-b-711318-675db99787-mk5zv\" (UID: \"6421252b-76d2-40ed-af4c-9e8086e8b87b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv" Nov 03 02:28:57.746423 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.746400 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv" Nov 03 02:28:57.863825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:57.863778 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:58.124604 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:28:58.124572 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6421252b_76d2_40ed_af4c_9e8086e8b87b.slice/crio-ba771429317dc5b2b82a345a17c1778faa8a746e9533ea1eb3c615270cbd5c47 WatchSource:0}: Error finding container ba771429317dc5b2b82a345a17c1778faa8a746e9533ea1eb3c615270cbd5c47: Status 404 returned error can't find the container with id ba771429317dc5b2b82a345a17c1778faa8a746e9533ea1eb3c615270cbd5c47 Nov 03 02:28:58.127140 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:58.127115 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv"] Nov 03 02:28:58.367706 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:58.367662 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:58.512159 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:58.512116 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:58.771313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:58.771222 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv" event={"ID":"6421252b-76d2-40ed-af4c-9e8086e8b87b","Type":"ContainerStarted","Data":"0f81047507d04a713129c101a7eac62aa71337fc9959de90b76020d3aea4342f"} Nov 03 02:28:58.771313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:58.771283 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv" event={"ID":"6421252b-76d2-40ed-af4c-9e8086e8b87b","Type":"ContainerStarted","Data":"ba771429317dc5b2b82a345a17c1778faa8a746e9533ea1eb3c615270cbd5c47"} Nov 03 02:28:58.771614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:58.771428 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv" Nov 03 02:28:58.886367 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:58.886298 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv" podStartSLOduration=1.886283234 podStartE2EDuration="1.886283234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:28:58.884508644 +0000 UTC m=+2220.568617800" watchObservedRunningTime="2026-11-03 02:28:58.886283234 +0000 UTC m=+2220.570392389" Nov 03 02:28:59.243406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:59.243335 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:59.402213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:59.402156 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:59.547002 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:28:59.546911 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:00.250749 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:00.250722 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw_9c92a303-5b7f-438e-8839-6ae9148d340a/payload-processing/0.log" Nov 03 02:29:01.175620 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:01.175593 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s_2063d869-2397-4105-bdb3-ea049d4bd196/payload-processing/0.log" Nov 03 02:29:01.386375 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:01.386343 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s_2063d869-2397-4105-bdb3-ea049d4bd196/payload-processing/0.log" Nov 03 02:29:02.464348 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:02.464318 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:03.489654 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:03.489619 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:03.770918 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:03.770825 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:05.630853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:05.630811 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:07.408234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:07.408175 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm"] Nov 03 02:29:07.498237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:07.498202 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:09.166243 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:09.166208 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:09.326061 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:09.326030 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:09.782542 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:09.782508 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv" Nov 03 02:29:10.718679 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:10.718643 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:11.229353 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:11.229322 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz"] Nov 03 02:29:11.229537 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:11.229516 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" podUID="cbecd0cc-1384-46b8-8eaa-758627373292" containerName="istio-proxy" containerID="cri-o://c7a1a56fbf01f2ed0f3e06f56d0b5a699859e7a3a02a25b8c8c6bcc6006ba227" gracePeriod=30 Nov 03 02:29:11.241746 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:11.241722 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:12.313966 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:12.313930 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:13.009176 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:29:13.009141 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:29:14.185662 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:14.185627 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:15.593866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:15.593829 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:16.520789 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.520763 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:29:16.659975 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.659942 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbecd0cc-1384-46b8-8eaa-758627373292-istio-token\" (UniqueName: \"kubernetes.io/projected/cbecd0cc-1384-46b8-8eaa-758627373292-istio-token\") pod \"cbecd0cc-1384-46b8-8eaa-758627373292\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " Nov 03 02:29:16.660489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.659983 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-workload-certs\") pod \"cbecd0cc-1384-46b8-8eaa-758627373292\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " Nov 03 02:29:16.660489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.660013 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-credential-socket\") pod \"cbecd0cc-1384-46b8-8eaa-758627373292\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " Nov 03 02:29:16.660489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.660055 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-workload-socket\") pod \"cbecd0cc-1384-46b8-8eaa-758627373292\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " Nov 03 02:29:16.660489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.660081 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-istio-envoy\") pod \"cbecd0cc-1384-46b8-8eaa-758627373292\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " Nov 03 02:29:16.660489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.660130 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbecd0cc-1384-46b8-8eaa-758627373292-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbecd0cc-1384-46b8-8eaa-758627373292-istiod-ca-cert\") pod \"cbecd0cc-1384-46b8-8eaa-758627373292\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " Nov 03 02:29:16.660489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.660170 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-istio-data\") pod \"cbecd0cc-1384-46b8-8eaa-758627373292\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " Nov 03 02:29:16.660489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.660221 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbecd0cc-1384-46b8-8eaa-758627373292-kube-api-access-76j7f\" (UniqueName: \"kubernetes.io/projected/cbecd0cc-1384-46b8-8eaa-758627373292-kube-api-access-76j7f\") pod \"cbecd0cc-1384-46b8-8eaa-758627373292\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " Nov 03 02:29:16.660489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.660270 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbecd0cc-1384-46b8-8eaa-758627373292-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbecd0cc-1384-46b8-8eaa-758627373292-istio-podinfo\") pod \"cbecd0cc-1384-46b8-8eaa-758627373292\" (UID: \"cbecd0cc-1384-46b8-8eaa-758627373292\") " Nov 03 02:29:16.660489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.660325 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-workload-certs" pod "cbecd0cc-1384-46b8-8eaa-758627373292" (UID: "cbecd0cc-1384-46b8-8eaa-758627373292"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:29:16.661017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.660498 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-istio-data" pod "cbecd0cc-1384-46b8-8eaa-758627373292" (UID: "cbecd0cc-1384-46b8-8eaa-758627373292"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:29:16.661017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.660760 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-workload-socket" pod "cbecd0cc-1384-46b8-8eaa-758627373292" (UID: "cbecd0cc-1384-46b8-8eaa-758627373292"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:29:16.661017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.660818 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-credential-socket" pod "cbecd0cc-1384-46b8-8eaa-758627373292" (UID: "cbecd0cc-1384-46b8-8eaa-758627373292"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:29:16.661017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.660823 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:16.661017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.660846 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbecd0cc-1384-46b8-8eaa-758627373292-istiod-ca-cert" pod "cbecd0cc-1384-46b8-8eaa-758627373292" (UID: "cbecd0cc-1384-46b8-8eaa-758627373292"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:29:16.661017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.660859 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:16.662918 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.662885 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbecd0cc-1384-46b8-8eaa-758627373292-istio-token" pod "cbecd0cc-1384-46b8-8eaa-758627373292" (UID: "cbecd0cc-1384-46b8-8eaa-758627373292"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:29:16.662918 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.662909 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbecd0cc-1384-46b8-8eaa-758627373292-istio-podinfo" pod "cbecd0cc-1384-46b8-8eaa-758627373292" (UID: "cbecd0cc-1384-46b8-8eaa-758627373292"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:29:16.663092 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.663060 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbecd0cc-1384-46b8-8eaa-758627373292-kube-api-access-76j7f" pod "cbecd0cc-1384-46b8-8eaa-758627373292" (UID: "cbecd0cc-1384-46b8-8eaa-758627373292"). InnerVolumeSpecName "kube-api-access-76j7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:29:16.663251 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.663233 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-istio-envoy" pod "cbecd0cc-1384-46b8-8eaa-758627373292" (UID: "cbecd0cc-1384-46b8-8eaa-758627373292"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:29:16.762481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.762392 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbecd0cc-1384-46b8-8eaa-758627373292-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:16.762481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.762424 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:16.762481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.762437 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:16.762481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.762448 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbecd0cc-1384-46b8-8eaa-758627373292-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:16.762481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.762461 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbecd0cc-1384-46b8-8eaa-758627373292-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:16.762481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.762474 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76j7f\" (UniqueName: \"kubernetes.io/projected/cbecd0cc-1384-46b8-8eaa-758627373292-kube-api-access-76j7f\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:16.762809 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.762489 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbecd0cc-1384-46b8-8eaa-758627373292-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:16.909771 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.909741 5463 generic.go:356] "Generic (PLEG): container finished" podID="cbecd0cc-1384-46b8-8eaa-758627373292" containerID="c7a1a56fbf01f2ed0f3e06f56d0b5a699859e7a3a02a25b8c8c6bcc6006ba227" exitCode=0 Nov 03 02:29:16.909979 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.909798 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" Nov 03 02:29:16.909979 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.909826 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" event={"ID":"cbecd0cc-1384-46b8-8eaa-758627373292","Type":"ContainerDied","Data":"c7a1a56fbf01f2ed0f3e06f56d0b5a699859e7a3a02a25b8c8c6bcc6006ba227"} Nov 03 02:29:16.909979 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.909887 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz" event={"ID":"cbecd0cc-1384-46b8-8eaa-758627373292","Type":"ContainerDied","Data":"c4d58b94df91c19f42126193a888fd749f3c4dc1912c3979a866d81a55dc5b96"} Nov 03 02:29:16.909979 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.909910 5463 scope.go:122] "RemoveContainer" containerID="c7a1a56fbf01f2ed0f3e06f56d0b5a699859e7a3a02a25b8c8c6bcc6006ba227" Nov 03 02:29:16.921379 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.921360 5463 scope.go:122] "RemoveContainer" containerID="fd2edee22c38d95c9e6e3273e986c4ecd2c743c1efeb7c94a413853922eb347c" Nov 03 02:29:16.931141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.931124 5463 scope.go:122] "RemoveContainer" containerID="c7a1a56fbf01f2ed0f3e06f56d0b5a699859e7a3a02a25b8c8c6bcc6006ba227" Nov 03 02:29:16.931409 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:29:16.931392 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a1a56fbf01f2ed0f3e06f56d0b5a699859e7a3a02a25b8c8c6bcc6006ba227\": container with ID starting with c7a1a56fbf01f2ed0f3e06f56d0b5a699859e7a3a02a25b8c8c6bcc6006ba227 not found: ID does not exist" containerID="c7a1a56fbf01f2ed0f3e06f56d0b5a699859e7a3a02a25b8c8c6bcc6006ba227" Nov 03 02:29:16.931521 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.931419 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a1a56fbf01f2ed0f3e06f56d0b5a699859e7a3a02a25b8c8c6bcc6006ba227"} err="failed to get container status \"c7a1a56fbf01f2ed0f3e06f56d0b5a699859e7a3a02a25b8c8c6bcc6006ba227\": rpc error: code = NotFound desc = could not find container \"c7a1a56fbf01f2ed0f3e06f56d0b5a699859e7a3a02a25b8c8c6bcc6006ba227\": container with ID starting with c7a1a56fbf01f2ed0f3e06f56d0b5a699859e7a3a02a25b8c8c6bcc6006ba227 not found: ID does not exist" Nov 03 02:29:16.931521 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.931441 5463 scope.go:122] "RemoveContainer" containerID="fd2edee22c38d95c9e6e3273e986c4ecd2c743c1efeb7c94a413853922eb347c" Nov 03 02:29:16.931685 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:29:16.931667 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd2edee22c38d95c9e6e3273e986c4ecd2c743c1efeb7c94a413853922eb347c\": container with ID starting with fd2edee22c38d95c9e6e3273e986c4ecd2c743c1efeb7c94a413853922eb347c not found: ID does not exist" containerID="fd2edee22c38d95c9e6e3273e986c4ecd2c743c1efeb7c94a413853922eb347c" Nov 03 02:29:16.931722 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:16.931691 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd2edee22c38d95c9e6e3273e986c4ecd2c743c1efeb7c94a413853922eb347c"} err="failed to get container status \"fd2edee22c38d95c9e6e3273e986c4ecd2c743c1efeb7c94a413853922eb347c\": rpc error: code = NotFound desc = could not find container \"fd2edee22c38d95c9e6e3273e986c4ecd2c743c1efeb7c94a413853922eb347c\": container with ID starting with fd2edee22c38d95c9e6e3273e986c4ecd2c743c1efeb7c94a413853922eb347c not found: ID does not exist" Nov 03 02:29:17.076020 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:17.075941 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz"] Nov 03 02:29:17.079169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:17.079144 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-7f9d9d-openshift-default-667d74574f-xsqzz"] Nov 03 02:29:17.228162 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:17.228130 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:18.444592 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:18.444562 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbecd0cc-1384-46b8-8eaa-758627373292" path="/var/lib/kubelet/pods/cbecd0cc-1384-46b8-8eaa-758627373292/volumes" Nov 03 02:29:22.119593 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.119561 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:22.429254 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.429215 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" podUID="aa346df3-a524-419b-b3a4-abf4ced623c0" containerName="main" containerID="cri-o://fae2b633443e001b21d5f313be963a91102dc7a46c22352077a83f4360fc72cd" gracePeriod=45 Nov 03 02:29:22.705658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.705632 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:29:22.740206 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.740109 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-tmp-dir\") pod \"aa346df3-a524-419b-b3a4-abf4ced623c0\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " Nov 03 02:29:22.740206 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.740161 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa346df3-a524-419b-b3a4-abf4ced623c0-kube-api-access-7sqnm\" (UniqueName: \"kubernetes.io/projected/aa346df3-a524-419b-b3a4-abf4ced623c0-kube-api-access-7sqnm\") pod \"aa346df3-a524-419b-b3a4-abf4ced623c0\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " Nov 03 02:29:22.740471 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.740298 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa346df3-a524-419b-b3a4-abf4ced623c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa346df3-a524-419b-b3a4-abf4ced623c0-tls-certs\") pod \"aa346df3-a524-419b-b3a4-abf4ced623c0\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " Nov 03 02:29:22.740471 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.740363 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-home\" (UniqueName: \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-home\") pod \"aa346df3-a524-419b-b3a4-abf4ced623c0\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " Nov 03 02:29:22.740471 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.740433 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-model-cache\") pod \"aa346df3-a524-419b-b3a4-abf4ced623c0\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " Nov 03 02:29:22.740471 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.740466 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-dshm\") pod \"aa346df3-a524-419b-b3a4-abf4ced623c0\" (UID: \"aa346df3-a524-419b-b3a4-abf4ced623c0\") " Nov 03 02:29:22.742423 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.742377 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-tmp-dir" pod "aa346df3-a524-419b-b3a4-abf4ced623c0" (UID: "aa346df3-a524-419b-b3a4-abf4ced623c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:29:22.742978 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.742905 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-model-cache" pod "aa346df3-a524-419b-b3a4-abf4ced623c0" (UID: "aa346df3-a524-419b-b3a4-abf4ced623c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:29:22.742978 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.742958 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-home" pod "aa346df3-a524-419b-b3a4-abf4ced623c0" (UID: "aa346df3-a524-419b-b3a4-abf4ced623c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:29:22.746237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.746175 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-dshm" pod "aa346df3-a524-419b-b3a4-abf4ced623c0" (UID: "aa346df3-a524-419b-b3a4-abf4ced623c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:29:22.747988 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.747959 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa346df3-a524-419b-b3a4-abf4ced623c0-tls-certs" pod "aa346df3-a524-419b-b3a4-abf4ced623c0" (UID: "aa346df3-a524-419b-b3a4-abf4ced623c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:29:22.748112 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.747991 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa346df3-a524-419b-b3a4-abf4ced623c0-kube-api-access-7sqnm" pod "aa346df3-a524-419b-b3a4-abf4ced623c0" (UID: "aa346df3-a524-419b-b3a4-abf4ced623c0"). InnerVolumeSpecName "kube-api-access-7sqnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:29:22.842096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.842064 5463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-tmp-dir\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:22.842096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.842094 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sqnm\" (UniqueName: \"kubernetes.io/projected/aa346df3-a524-419b-b3a4-abf4ced623c0-kube-api-access-7sqnm\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:22.842345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.842111 5463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa346df3-a524-419b-b3a4-abf4ced623c0-tls-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:22.842345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.842122 5463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-home\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:22.842345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.842134 5463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-model-cache\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:22.842345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.842145 5463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa346df3-a524-419b-b3a4-abf4ced623c0-dshm\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:22.987810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.987149 5463 generic.go:356] "Generic (PLEG): container finished" podID="aa346df3-a524-419b-b3a4-abf4ced623c0" containerID="fae2b633443e001b21d5f313be963a91102dc7a46c22352077a83f4360fc72cd" exitCode=0 Nov 03 02:29:22.987810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.987262 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" Nov 03 02:29:22.987810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.987525 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" event={"ID":"aa346df3-a524-419b-b3a4-abf4ced623c0","Type":"ContainerDied","Data":"fae2b633443e001b21d5f313be963a91102dc7a46c22352077a83f4360fc72cd"} Nov 03 02:29:22.987810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.987570 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm" event={"ID":"aa346df3-a524-419b-b3a4-abf4ced623c0","Type":"ContainerDied","Data":"34c9bdf7b82676f02b198d82951c16df2b2488d2cf8f961e573a1aa31d3feb80"} Nov 03 02:29:22.987810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:22.987596 5463 scope.go:122] "RemoveContainer" containerID="fae2b633443e001b21d5f313be963a91102dc7a46c22352077a83f4360fc72cd" Nov 03 02:29:23.010851 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:23.010828 5463 scope.go:122] "RemoveContainer" containerID="fae2b633443e001b21d5f313be963a91102dc7a46c22352077a83f4360fc72cd" Nov 03 02:29:23.011257 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:29:23.011225 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fae2b633443e001b21d5f313be963a91102dc7a46c22352077a83f4360fc72cd\": container with ID starting with fae2b633443e001b21d5f313be963a91102dc7a46c22352077a83f4360fc72cd not found: ID does not exist" containerID="fae2b633443e001b21d5f313be963a91102dc7a46c22352077a83f4360fc72cd" Nov 03 02:29:23.011395 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:23.011262 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fae2b633443e001b21d5f313be963a91102dc7a46c22352077a83f4360fc72cd"} err="failed to get container status \"fae2b633443e001b21d5f313be963a91102dc7a46c22352077a83f4360fc72cd\": rpc error: code = NotFound desc = could not find container \"fae2b633443e001b21d5f313be963a91102dc7a46c22352077a83f4360fc72cd\": container with ID starting with fae2b633443e001b21d5f313be963a91102dc7a46c22352077a83f4360fc72cd not found: ID does not exist" Nov 03 02:29:23.152853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:23.152822 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm"] Nov 03 02:29:23.156477 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:23.156454 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-56299c/ipp-route-56299c-kserve-976d85fc-fl7fm"] Nov 03 02:29:23.220652 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:23.220623 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:23.667701 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:23.667669 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824"] Nov 03 02:29:23.775363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:23.775332 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq"] Nov 03 02:29:24.037780 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.037701 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:24.190584 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.190554 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:24.444506 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.444475 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa346df3-a524-419b-b3a4-abf4ced623c0" path="/var/lib/kubelet/pods/aa346df3-a524-419b-b3a4-abf4ced623c0/volumes" Nov 03 02:29:24.618969 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.618939 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl"] Nov 03 02:29:24.623352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.620242 5463 cpu_manager.go:414] "containerMap: removing container" podUID="aa346df3-a524-419b-b3a4-abf4ced623c0" containerName="main" Nov 03 02:29:24.623352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.620273 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa346df3-a524-419b-b3a4-abf4ced623c0" containerName="main" Nov 03 02:29:24.623352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.620331 5463 cpu_manager.go:414] "containerMap: removing container" podUID="cbecd0cc-1384-46b8-8eaa-758627373292" containerName="istio-proxy" Nov 03 02:29:24.623352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.620340 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbecd0cc-1384-46b8-8eaa-758627373292" containerName="istio-proxy" Nov 03 02:29:24.623352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.620433 5463 cpu_manager.go:414] "containerMap: removing container" podUID="cbecd0cc-1384-46b8-8eaa-758627373292" containerName="istio-proxy" Nov 03 02:29:24.623352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.620440 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbecd0cc-1384-46b8-8eaa-758627373292" containerName="istio-proxy" Nov 03 02:29:24.623352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.620890 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa346df3-a524-419b-b3a4-abf4ced623c0" containerName="main" Nov 03 02:29:24.623352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.620906 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8d34a09-e4c1-4298-b62d-7dec1e3f9e1f" containerName="istio-proxy" Nov 03 02:29:24.623352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.620921 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbecd0cc-1384-46b8-8eaa-758627373292" containerName="istio-proxy" Nov 03 02:29:24.623352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.620951 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbecd0cc-1384-46b8-8eaa-758627373292" containerName="istio-proxy" Nov 03 02:29:24.628279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.628259 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.672533 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.672495 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.672733 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.672552 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.672733 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.672704 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.672869 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.672744 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.672869 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.672773 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.672869 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.672841 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.673008 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.672892 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.673008 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.672932 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.673075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.673047 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdccg\" (UniqueName: \"kubernetes.io/projected/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-kube-api-access-fdccg\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.679067 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.679040 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl"] Nov 03 02:29:24.681902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.681877 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f9bff9-openshift-default-dockercfg-nn2lt\"" Nov 03 02:29:24.727807 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.727721 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:24.798246 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.791661 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.802212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.791754 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.802212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.799062 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.802212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.799219 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdccg\" (UniqueName: \"kubernetes.io/projected/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-kube-api-access-fdccg\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.802212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.799293 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.802212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.799329 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.802212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.799390 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.802212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.799420 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.802212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.799451 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.802212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.800320 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.802212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.800665 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.842228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.821148 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.842228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.821695 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.842228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.822729 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.842228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.825048 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.842228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.825641 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.879302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.879171 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.879302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.879231 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdccg\" (UniqueName: \"kubernetes.io/projected/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-kube-api-access-fdccg\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:24.945330 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:24.945296 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:25.070956 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:25.070922 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:25.371425 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:29:25.371394 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4fd375_fcd1_4665_8a2e_89a67bb9dc43.slice/crio-c9faedd5fb11c70f578e27684718889e0ccf3c1c895f58af5e3d02588ece6ed7 WatchSource:0}: Error finding container c9faedd5fb11c70f578e27684718889e0ccf3c1c895f58af5e3d02588ece6ed7: Status 404 returned error can't find the container with id c9faedd5fb11c70f578e27684718889e0ccf3c1c895f58af5e3d02588ece6ed7 Nov 03 02:29:25.374128 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:25.374102 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl"] Nov 03 02:29:25.374400 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:25.374372 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:29:25.374449 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:25.374433 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:29:25.374486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:25.374463 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:29:26.048680 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:26.048646 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" event={"ID":"da4fd375-fcd1-4665-8a2e-89a67bb9dc43","Type":"ContainerStarted","Data":"4b6e748def28d2bc8156b0730cc50284d393f153af9ed4bee58ea6d6050e1dbf"} Nov 03 02:29:26.048866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:26.048696 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" event={"ID":"da4fd375-fcd1-4665-8a2e-89a67bb9dc43","Type":"ContainerStarted","Data":"c9faedd5fb11c70f578e27684718889e0ccf3c1c895f58af5e3d02588ece6ed7"} Nov 03 02:29:26.269800 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:26.269732 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" podStartSLOduration=2.269711895 podStartE2EDuration="2.269711895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:29:26.217777659 +0000 UTC m=+2247.901886807" watchObservedRunningTime="2026-11-03 02:29:26.269711895 +0000 UTC m=+2247.953821051" Nov 03 02:29:26.945960 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:26.945920 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:26.952372 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:26.952346 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:27.057886 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:27.057854 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:27.058905 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:27.058882 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:29:28.668646 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:28.668592 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824" podUID="839fcc2f-d52f-48fe-be5c-918dd68526f4" containerName="payload-processing" containerID="cri-o://4ffa72d3b14e65b755835dc77fbd2872d4c9b4da43f8f27cdaab5b3054e80849" gracePeriod=25 Nov 03 02:29:28.776165 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:28.776120 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq" podUID="3d7216d2-aafe-4501-9085-38ee727a1b97" containerName="payload-pre-processing" containerID="cri-o://a6da390b481538b088d0507401371d97e5dbe3d0fcef24b0542bf71e6cecb636" gracePeriod=25 Nov 03 02:29:28.962758 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:28.962448 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824" Nov 03 02:29:29.054486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.054455 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq" Nov 03 02:29:29.063342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.063319 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/839fcc2f-d52f-48fe-be5c-918dd68526f4-kube-api-access-2p7tv\" (UniqueName: \"kubernetes.io/projected/839fcc2f-d52f-48fe-be5c-918dd68526f4-kube-api-access-2p7tv\") pod \"839fcc2f-d52f-48fe-be5c-918dd68526f4\" (UID: \"839fcc2f-d52f-48fe-be5c-918dd68526f4\") " Nov 03 02:29:29.063539 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.063521 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/839fcc2f-d52f-48fe-be5c-918dd68526f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839fcc2f-d52f-48fe-be5c-918dd68526f4-plugins-config-volume\") pod \"839fcc2f-d52f-48fe-be5c-918dd68526f4\" (UID: \"839fcc2f-d52f-48fe-be5c-918dd68526f4\") " Nov 03 02:29:29.063910 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.063877 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839fcc2f-d52f-48fe-be5c-918dd68526f4-plugins-config-volume" pod "839fcc2f-d52f-48fe-be5c-918dd68526f4" (UID: "839fcc2f-d52f-48fe-be5c-918dd68526f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:29:29.065568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.065539 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839fcc2f-d52f-48fe-be5c-918dd68526f4-kube-api-access-2p7tv" pod "839fcc2f-d52f-48fe-be5c-918dd68526f4" (UID: "839fcc2f-d52f-48fe-be5c-918dd68526f4"). InnerVolumeSpecName "kube-api-access-2p7tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:29:29.076407 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.076365 5463 generic.go:356] "Generic (PLEG): container finished" podID="3d7216d2-aafe-4501-9085-38ee727a1b97" containerID="a6da390b481538b088d0507401371d97e5dbe3d0fcef24b0542bf71e6cecb636" exitCode=0 Nov 03 02:29:29.076613 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.076591 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq" event={"ID":"3d7216d2-aafe-4501-9085-38ee727a1b97","Type":"ContainerDied","Data":"a6da390b481538b088d0507401371d97e5dbe3d0fcef24b0542bf71e6cecb636"} Nov 03 02:29:29.076701 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.076649 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq" event={"ID":"3d7216d2-aafe-4501-9085-38ee727a1b97","Type":"ContainerDied","Data":"b1b893d268d7b6aa64a15e12562e5c2d1cf4f892aa8f6c64d656d16f2792c14d"} Nov 03 02:29:29.076701 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.076681 5463 scope.go:122] "RemoveContainer" containerID="a6da390b481538b088d0507401371d97e5dbe3d0fcef24b0542bf71e6cecb636" Nov 03 02:29:29.076894 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.076858 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq" Nov 03 02:29:29.091544 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.091513 5463 generic.go:356] "Generic (PLEG): container finished" podID="839fcc2f-d52f-48fe-be5c-918dd68526f4" containerID="4ffa72d3b14e65b755835dc77fbd2872d4c9b4da43f8f27cdaab5b3054e80849" exitCode=0 Nov 03 02:29:29.092074 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.092054 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824" Nov 03 02:29:29.092552 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.092529 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824" event={"ID":"839fcc2f-d52f-48fe-be5c-918dd68526f4","Type":"ContainerDied","Data":"4ffa72d3b14e65b755835dc77fbd2872d4c9b4da43f8f27cdaab5b3054e80849"} Nov 03 02:29:29.092655 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.092580 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824" event={"ID":"839fcc2f-d52f-48fe-be5c-918dd68526f4","Type":"ContainerDied","Data":"bc2f3306f422bc50e8dae95bb94cd7002849135fdd20f0d5c606821df2a15420"} Nov 03 02:29:29.093642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.093623 5463 scope.go:122] "RemoveContainer" containerID="a6da390b481538b088d0507401371d97e5dbe3d0fcef24b0542bf71e6cecb636" Nov 03 02:29:29.093978 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:29:29.093944 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6da390b481538b088d0507401371d97e5dbe3d0fcef24b0542bf71e6cecb636\": container with ID starting with a6da390b481538b088d0507401371d97e5dbe3d0fcef24b0542bf71e6cecb636 not found: ID does not exist" containerID="a6da390b481538b088d0507401371d97e5dbe3d0fcef24b0542bf71e6cecb636" Nov 03 02:29:29.094055 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.093990 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6da390b481538b088d0507401371d97e5dbe3d0fcef24b0542bf71e6cecb636"} err="failed to get container status \"a6da390b481538b088d0507401371d97e5dbe3d0fcef24b0542bf71e6cecb636\": rpc error: code = NotFound desc = could not find container \"a6da390b481538b088d0507401371d97e5dbe3d0fcef24b0542bf71e6cecb636\": container with ID starting with a6da390b481538b088d0507401371d97e5dbe3d0fcef24b0542bf71e6cecb636 not found: ID does not exist" Nov 03 02:29:29.094055 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.094012 5463 scope.go:122] "RemoveContainer" containerID="4ffa72d3b14e65b755835dc77fbd2872d4c9b4da43f8f27cdaab5b3054e80849" Nov 03 02:29:29.105581 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.105558 5463 scope.go:122] "RemoveContainer" containerID="4ffa72d3b14e65b755835dc77fbd2872d4c9b4da43f8f27cdaab5b3054e80849" Nov 03 02:29:29.105896 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:29:29.105872 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ffa72d3b14e65b755835dc77fbd2872d4c9b4da43f8f27cdaab5b3054e80849\": container with ID starting with 4ffa72d3b14e65b755835dc77fbd2872d4c9b4da43f8f27cdaab5b3054e80849 not found: ID does not exist" containerID="4ffa72d3b14e65b755835dc77fbd2872d4c9b4da43f8f27cdaab5b3054e80849" Nov 03 02:29:29.105991 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.105904 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ffa72d3b14e65b755835dc77fbd2872d4c9b4da43f8f27cdaab5b3054e80849"} err="failed to get container status \"4ffa72d3b14e65b755835dc77fbd2872d4c9b4da43f8f27cdaab5b3054e80849\": rpc error: code = NotFound desc = could not find container \"4ffa72d3b14e65b755835dc77fbd2872d4c9b4da43f8f27cdaab5b3054e80849\": container with ID starting with 4ffa72d3b14e65b755835dc77fbd2872d4c9b4da43f8f27cdaab5b3054e80849 not found: ID does not exist" Nov 03 02:29:29.158011 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.157955 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:29.165126 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.165102 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d7216d2-aafe-4501-9085-38ee727a1b97-kube-api-access-8t6tk\" (UniqueName: \"kubernetes.io/projected/3d7216d2-aafe-4501-9085-38ee727a1b97-kube-api-access-8t6tk\") pod \"3d7216d2-aafe-4501-9085-38ee727a1b97\" (UID: \"3d7216d2-aafe-4501-9085-38ee727a1b97\") " Nov 03 02:29:29.165302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.165158 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d7216d2-aafe-4501-9085-38ee727a1b97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d7216d2-aafe-4501-9085-38ee727a1b97-plugins-config-volume\") pod \"3d7216d2-aafe-4501-9085-38ee727a1b97\" (UID: \"3d7216d2-aafe-4501-9085-38ee727a1b97\") " Nov 03 02:29:29.165589 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.165561 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d7216d2-aafe-4501-9085-38ee727a1b97-plugins-config-volume" pod "3d7216d2-aafe-4501-9085-38ee727a1b97" (UID: "3d7216d2-aafe-4501-9085-38ee727a1b97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:29:29.166165 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.166139 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p7tv\" (UniqueName: \"kubernetes.io/projected/839fcc2f-d52f-48fe-be5c-918dd68526f4-kube-api-access-2p7tv\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:29.166282 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.166165 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839fcc2f-d52f-48fe-be5c-918dd68526f4-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:29.166282 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.166179 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d7216d2-aafe-4501-9085-38ee727a1b97-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:29.167709 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.167689 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7216d2-aafe-4501-9085-38ee727a1b97-kube-api-access-8t6tk" pod "3d7216d2-aafe-4501-9085-38ee727a1b97" (UID: "3d7216d2-aafe-4501-9085-38ee727a1b97"). InnerVolumeSpecName "kube-api-access-8t6tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:29:29.267425 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.267346 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t6tk\" (UniqueName: \"kubernetes.io/projected/3d7216d2-aafe-4501-9085-38ee727a1b97-kube-api-access-8t6tk\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:29.305531 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.305502 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824"] Nov 03 02:29:29.309824 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.309799 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb1940-5f8d57b6bf-2n824"] Nov 03 02:29:29.498426 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.498393 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq"] Nov 03 02:29:29.501128 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.501110 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-5c58f78587-x7gvq"] Nov 03 02:29:29.899468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:29.899437 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:30.072875 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:30.072793 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:30.230955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:30.230908 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:30.444739 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:30.444710 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d7216d2-aafe-4501-9085-38ee727a1b97" path="/var/lib/kubelet/pods/3d7216d2-aafe-4501-9085-38ee727a1b97/volumes" Nov 03 02:29:30.445155 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:30.445142 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="839fcc2f-d52f-48fe-be5c-918dd68526f4" path="/var/lib/kubelet/pods/839fcc2f-d52f-48fe-be5c-918dd68526f4/volumes" Nov 03 02:29:33.109203 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.109139 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:33.291526 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.291496 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:33.504360 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.504321 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b"] Nov 03 02:29:33.504943 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.504929 5463 cpu_manager.go:414] "containerMap: removing container" podUID="839fcc2f-d52f-48fe-be5c-918dd68526f4" containerName="payload-processing" Nov 03 02:29:33.504990 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.504945 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839fcc2f-d52f-48fe-be5c-918dd68526f4" containerName="payload-processing" Nov 03 02:29:33.504990 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.504962 5463 cpu_manager.go:414] "containerMap: removing container" podUID="3d7216d2-aafe-4501-9085-38ee727a1b97" containerName="payload-pre-processing" Nov 03 02:29:33.504990 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.504967 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7216d2-aafe-4501-9085-38ee727a1b97" containerName="payload-pre-processing" Nov 03 02:29:33.505093 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.505061 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="839fcc2f-d52f-48fe-be5c-918dd68526f4" containerName="payload-processing" Nov 03 02:29:33.505093 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.505070 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7216d2-aafe-4501-9085-38ee727a1b97" containerName="payload-pre-processing" Nov 03 02:29:33.510055 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.510030 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b" Nov 03 02:29:33.562270 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.562238 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b"] Nov 03 02:29:33.612589 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.612550 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f9bff9-dockercfg-bwkvq\"" Nov 03 02:29:33.617512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.612936 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f9bff9\"" Nov 03 02:29:33.631164 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.631130 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763ef8e4-77b4-4e99-8bf6-839d76b88c99-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b\" (UID: \"763ef8e4-77b4-4e99-8bf6-839d76b88c99\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b" Nov 03 02:29:33.631547 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.631515 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlvzx\" (UniqueName: \"kubernetes.io/projected/763ef8e4-77b4-4e99-8bf6-839d76b88c99-kube-api-access-xlvzx\") pod \"payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b\" (UID: \"763ef8e4-77b4-4e99-8bf6-839d76b88c99\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b" Nov 03 02:29:33.666368 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.666339 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw"] Nov 03 02:29:33.670715 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.670692 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw" Nov 03 02:29:33.718912 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.718885 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw"] Nov 03 02:29:33.732869 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.732838 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763ef8e4-77b4-4e99-8bf6-839d76b88c99-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b\" (UID: \"763ef8e4-77b4-4e99-8bf6-839d76b88c99\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b" Nov 03 02:29:33.733024 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.732928 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlvzx\" (UniqueName: \"kubernetes.io/projected/763ef8e4-77b4-4e99-8bf6-839d76b88c99-kube-api-access-xlvzx\") pod \"payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b\" (UID: \"763ef8e4-77b4-4e99-8bf6-839d76b88c99\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b" Nov 03 02:29:33.733600 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.733578 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763ef8e4-77b4-4e99-8bf6-839d76b88c99-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b\" (UID: \"763ef8e4-77b4-4e99-8bf6-839d76b88c99\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b" Nov 03 02:29:33.789553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.789487 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlvzx\" (UniqueName: \"kubernetes.io/projected/763ef8e4-77b4-4e99-8bf6-839d76b88c99-kube-api-access-xlvzx\") pod \"payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b\" (UID: \"763ef8e4-77b4-4e99-8bf6-839d76b88c99\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b" Nov 03 02:29:33.825055 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.825029 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b" Nov 03 02:29:33.834380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.834356 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2q9k\" (UniqueName: \"kubernetes.io/projected/06d24e51-c948-4599-b7db-1ea7082bc879-kube-api-access-w2q9k\") pod \"payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw\" (UID: \"06d24e51-c948-4599-b7db-1ea7082bc879\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw" Nov 03 02:29:33.834567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.834547 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d24e51-c948-4599-b7db-1ea7082bc879-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw\" (UID: \"06d24e51-c948-4599-b7db-1ea7082bc879\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw" Nov 03 02:29:33.936130 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.936084 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d24e51-c948-4599-b7db-1ea7082bc879-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw\" (UID: \"06d24e51-c948-4599-b7db-1ea7082bc879\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw" Nov 03 02:29:33.936320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.936185 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2q9k\" (UniqueName: \"kubernetes.io/projected/06d24e51-c948-4599-b7db-1ea7082bc879-kube-api-access-w2q9k\") pod \"payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw\" (UID: \"06d24e51-c948-4599-b7db-1ea7082bc879\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw" Nov 03 02:29:33.937066 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.937040 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d24e51-c948-4599-b7db-1ea7082bc879-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw\" (UID: \"06d24e51-c948-4599-b7db-1ea7082bc879\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw" Nov 03 02:29:33.992946 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:33.992920 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2q9k\" (UniqueName: \"kubernetes.io/projected/06d24e51-c948-4599-b7db-1ea7082bc879-kube-api-access-w2q9k\") pod \"payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw\" (UID: \"06d24e51-c948-4599-b7db-1ea7082bc879\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw" Nov 03 02:29:34.194165 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:29:34.194124 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763ef8e4_77b4_4e99_8bf6_839d76b88c99.slice/crio-e7eb05fa13ef0c7be7876c619810d6a8982afce83cb99c2bd50e2d9168cfcac5 WatchSource:0}: Error finding container e7eb05fa13ef0c7be7876c619810d6a8982afce83cb99c2bd50e2d9168cfcac5: Status 404 returned error can't find the container with id e7eb05fa13ef0c7be7876c619810d6a8982afce83cb99c2bd50e2d9168cfcac5 Nov 03 02:29:34.197035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:34.197010 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b"] Nov 03 02:29:34.284367 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:34.284343 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw" Nov 03 02:29:34.521004 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:34.520932 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:34.655820 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:29:34.655779 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d24e51_c948_4599_b7db_1ea7082bc879.slice/crio-08bb222450938c6d7c20c682e0175b82b5bab70fa930db564986b7f8e5b109ac WatchSource:0}: Error finding container 08bb222450938c6d7c20c682e0175b82b5bab70fa930db564986b7f8e5b109ac: Status 404 returned error can't find the container with id 08bb222450938c6d7c20c682e0175b82b5bab70fa930db564986b7f8e5b109ac Nov 03 02:29:34.658937 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:34.658911 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw"] Nov 03 02:29:35.139394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:35.139355 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b" event={"ID":"763ef8e4-77b4-4e99-8bf6-839d76b88c99","Type":"ContainerStarted","Data":"7bb5fbbc0f922f137a5ac4f63923bf2287c25e792c4f11de492644fb55815ee7"} Nov 03 02:29:35.139619 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:35.139417 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b" event={"ID":"763ef8e4-77b4-4e99-8bf6-839d76b88c99","Type":"ContainerStarted","Data":"e7eb05fa13ef0c7be7876c619810d6a8982afce83cb99c2bd50e2d9168cfcac5"} Nov 03 02:29:35.139619 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:35.139557 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b" Nov 03 02:29:35.141042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:35.141015 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw" event={"ID":"06d24e51-c948-4599-b7db-1ea7082bc879","Type":"ContainerStarted","Data":"a261ce48fda2bc6384dadf9f6f99837552287e4c76eaea1948ab59c4381e6dee"} Nov 03 02:29:35.141161 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:35.141055 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw" event={"ID":"06d24e51-c948-4599-b7db-1ea7082bc879","Type":"ContainerStarted","Data":"08bb222450938c6d7c20c682e0175b82b5bab70fa930db564986b7f8e5b109ac"} Nov 03 02:29:35.141161 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:35.141098 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw" Nov 03 02:29:35.254898 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:35.254842 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b" podStartSLOduration=2.254828148 podStartE2EDuration="2.254828148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:29:35.251624953 +0000 UTC m=+2256.935734110" watchObservedRunningTime="2026-11-03 02:29:35.254828148 +0000 UTC m=+2256.938937302" Nov 03 02:29:35.368163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:35.368102 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw" podStartSLOduration=2.368084464 podStartE2EDuration="2.368084464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:29:35.366930187 +0000 UTC m=+2257.051039342" watchObservedRunningTime="2026-11-03 02:29:35.368084464 +0000 UTC m=+2257.052193620" Nov 03 02:29:39.439005 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:39.438923 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:43.008062 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:29:43.008024 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:29:44.240006 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:44.239967 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:44.549929 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:44.549843 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:44.721457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:44.721416 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:46.152232 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:46.152177 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b" Nov 03 02:29:46.152632 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:46.152446 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw" Nov 03 02:29:49.439990 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:49.439954 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:50.203973 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:50.203939 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:54.229254 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:54.229185 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:55.676912 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:55.676876 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:58.881463 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:58.881417 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:59.046710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:59.046670 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:59.372286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:29:59.372249 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:00.163226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.163160 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222"] Nov 03 02:30:00.167795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.167771 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" Nov 03 02:30:00.183019 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.182993 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl"] Nov 03 02:30:00.187001 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.186979 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl" Nov 03 02:30:00.188217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.188168 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25"] Nov 03 02:30:00.192438 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.192420 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25" Nov 03 02:30:00.214451 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.214426 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4"] Nov 03 02:30:00.218976 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.218957 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm"] Nov 03 02:30:00.219104 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.219073 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4" Nov 03 02:30:00.223401 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.223385 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm" Nov 03 02:30:00.260324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.260288 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222"] Nov 03 02:30:00.261311 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.261278 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk2vf\" (UniqueName: \"kubernetes.io/projected/0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98-kube-api-access-zk2vf\") pod \"maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm\" (UID: \"0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm" Nov 03 02:30:00.261479 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.261334 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c076195c-71f0-49cc-8aef-b690cab08678-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894550-tgdk4\" (UID: \"c076195c-71f0-49cc-8aef-b690cab08678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4" Nov 03 02:30:00.261479 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.261442 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rr9r\" (UniqueName: \"kubernetes.io/projected/cd0d24cd-c457-40b9-8786-378c54271667-kube-api-access-7rr9r\") pod \"maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222\" (UID: \"cd0d24cd-c457-40b9-8786-378c54271667\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" Nov 03 02:30:00.261584 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.261482 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25\" (UID: \"e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25" Nov 03 02:30:00.261624 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.261601 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd0d24cd-c457-40b9-8786-378c54271667-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222\" (UID: \"cd0d24cd-c457-40b9-8786-378c54271667\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" Nov 03 02:30:00.261669 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.261632 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvrtr\" (UniqueName: \"kubernetes.io/projected/c0c965f8-4f67-420c-9322-0afa9e745187-kube-api-access-jvrtr\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl\" (UID: \"c0c965f8-4f67-420c-9322-0afa9e745187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl" Nov 03 02:30:00.261706 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.261679 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm\" (UID: \"0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm" Nov 03 02:30:00.261797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.261775 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4kkl\" (UniqueName: \"kubernetes.io/projected/c076195c-71f0-49cc-8aef-b690cab08678-kube-api-access-d4kkl\") pod \"maas-api-key-cleanup-29894550-tgdk4\" (UID: \"c076195c-71f0-49cc-8aef-b690cab08678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4" Nov 03 02:30:00.261923 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.261826 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm2sw\" (UniqueName: \"kubernetes.io/projected/e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f-kube-api-access-wm2sw\") pod \"maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25\" (UID: \"e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25" Nov 03 02:30:00.261923 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.261908 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c965f8-4f67-420c-9322-0afa9e745187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl\" (UID: \"c0c965f8-4f67-420c-9322-0afa9e745187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl" Nov 03 02:30:00.314734 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.310599 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4"] Nov 03 02:30:00.314734 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.310999 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25"] Nov 03 02:30:00.315565 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.315539 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl"] Nov 03 02:30:00.317438 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.317411 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:30:00.362765 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.362736 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm\" (UID: \"0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm" Nov 03 02:30:00.362943 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.362809 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4kkl\" (UniqueName: \"kubernetes.io/projected/c076195c-71f0-49cc-8aef-b690cab08678-kube-api-access-d4kkl\") pod \"maas-api-key-cleanup-29894550-tgdk4\" (UID: \"c076195c-71f0-49cc-8aef-b690cab08678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4" Nov 03 02:30:00.362943 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.362832 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm2sw\" (UniqueName: \"kubernetes.io/projected/e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f-kube-api-access-wm2sw\") pod \"maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25\" (UID: \"e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25" Nov 03 02:30:00.362943 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.362848 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c965f8-4f67-420c-9322-0afa9e745187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl\" (UID: \"c0c965f8-4f67-420c-9322-0afa9e745187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl" Nov 03 02:30:00.362943 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.362903 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk2vf\" (UniqueName: \"kubernetes.io/projected/0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98-kube-api-access-zk2vf\") pod \"maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm\" (UID: \"0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm" Nov 03 02:30:00.363602 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.362943 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c076195c-71f0-49cc-8aef-b690cab08678-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894550-tgdk4\" (UID: \"c076195c-71f0-49cc-8aef-b690cab08678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4" Nov 03 02:30:00.363602 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.362976 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rr9r\" (UniqueName: \"kubernetes.io/projected/cd0d24cd-c457-40b9-8786-378c54271667-kube-api-access-7rr9r\") pod \"maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222\" (UID: \"cd0d24cd-c457-40b9-8786-378c54271667\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" Nov 03 02:30:00.363602 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.363007 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25\" (UID: \"e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25" Nov 03 02:30:00.363602 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.363131 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd0d24cd-c457-40b9-8786-378c54271667-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222\" (UID: \"cd0d24cd-c457-40b9-8786-378c54271667\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" Nov 03 02:30:00.363602 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.363164 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvrtr\" (UniqueName: \"kubernetes.io/projected/c0c965f8-4f67-420c-9322-0afa9e745187-kube-api-access-jvrtr\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl\" (UID: \"c0c965f8-4f67-420c-9322-0afa9e745187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl" Nov 03 02:30:00.364089 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.363913 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm\" (UID: \"0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm" Nov 03 02:30:00.364330 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.364308 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c076195c-71f0-49cc-8aef-b690cab08678-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894550-tgdk4\" (UID: \"c076195c-71f0-49cc-8aef-b690cab08678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4" Nov 03 02:30:00.364453 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.364436 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25\" (UID: \"e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25" Nov 03 02:30:00.364637 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.364616 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd0d24cd-c457-40b9-8786-378c54271667-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222\" (UID: \"cd0d24cd-c457-40b9-8786-378c54271667\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" Nov 03 02:30:00.364710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.364632 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c965f8-4f67-420c-9322-0afa9e745187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl\" (UID: \"c0c965f8-4f67-420c-9322-0afa9e745187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl" Nov 03 02:30:00.374239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.374075 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm"] Nov 03 02:30:00.421073 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.420987 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk2vf\" (UniqueName: \"kubernetes.io/projected/0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98-kube-api-access-zk2vf\") pod \"maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm\" (UID: \"0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm" Nov 03 02:30:00.421073 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.421064 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rr9r\" (UniqueName: \"kubernetes.io/projected/cd0d24cd-c457-40b9-8786-378c54271667-kube-api-access-7rr9r\") pod \"maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222\" (UID: \"cd0d24cd-c457-40b9-8786-378c54271667\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" Nov 03 02:30:00.421373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.421299 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4kkl\" (UniqueName: \"kubernetes.io/projected/c076195c-71f0-49cc-8aef-b690cab08678-kube-api-access-d4kkl\") pod \"maas-api-key-cleanup-29894550-tgdk4\" (UID: \"c076195c-71f0-49cc-8aef-b690cab08678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4" Nov 03 02:30:00.421437 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.421391 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvrtr\" (UniqueName: \"kubernetes.io/projected/c0c965f8-4f67-420c-9322-0afa9e745187-kube-api-access-jvrtr\") pod \"maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl\" (UID: \"c0c965f8-4f67-420c-9322-0afa9e745187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl" Nov 03 02:30:00.422171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.422154 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm2sw\" (UniqueName: \"kubernetes.io/projected/e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f-kube-api-access-wm2sw\") pod \"maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25\" (UID: \"e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25" Nov 03 02:30:00.484503 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.484474 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" Nov 03 02:30:00.508983 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.508748 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl" Nov 03 02:30:00.521209 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.517916 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25" Nov 03 02:30:00.534712 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.534687 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4" Nov 03 02:30:00.543915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.543887 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm" Nov 03 02:30:00.895646 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:30:00.895610 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0d24cd_c457_40b9_8786_378c54271667.slice/crio-f76d74b6a7856dfe2577c0858b2b507614c26f57d17c1211a3caf614bbd7a893 WatchSource:0}: Error finding container f76d74b6a7856dfe2577c0858b2b507614c26f57d17c1211a3caf614bbd7a893: Status 404 returned error can't find the container with id f76d74b6a7856dfe2577c0858b2b507614c26f57d17c1211a3caf614bbd7a893 Nov 03 02:30:00.898777 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.898755 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222"] Nov 03 02:30:00.946110 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.946084 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl"] Nov 03 02:30:00.950626 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.950603 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4"] Nov 03 02:30:00.953925 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.953903 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25"] Nov 03 02:30:00.995259 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:00.995231 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm"] Nov 03 02:30:01.006433 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:30:01.006408 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ecbfe03_1eaf_4c5b_9f8b_62d5c88e3c98.slice/crio-cea1e2e735ff93909640ded046677edba57d8bcdcdb842a3ed63a81c76d7a168 WatchSource:0}: Error finding container cea1e2e735ff93909640ded046677edba57d8bcdcdb842a3ed63a81c76d7a168: Status 404 returned error can't find the container with id cea1e2e735ff93909640ded046677edba57d8bcdcdb842a3ed63a81c76d7a168 Nov 03 02:30:01.155027 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.154998 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87"] Nov 03 02:30:01.160932 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.160908 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87" Nov 03 02:30:01.200447 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.200373 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjmbc\" (UniqueName: \"kubernetes.io/projected/bbd12200-ef29-4e84-966d-5faed0aca233-kube-api-access-zjmbc\") pod \"maas-api-revoke-sub-30d3e735-qqw87\" (UID: \"bbd12200-ef29-4e84-966d-5faed0aca233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87" Nov 03 02:30:01.200763 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.200450 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd12200-ef29-4e84-966d-5faed0aca233-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30d3e735-qqw87\" (UID: \"bbd12200-ef29-4e84-966d-5faed0aca233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87" Nov 03 02:30:01.200763 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.200591 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd12200-ef29-4e84-966d-5faed0aca233-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30d3e735-qqw87\" (UID: \"bbd12200-ef29-4e84-966d-5faed0aca233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87" Nov 03 02:30:01.231965 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.231933 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:30:01.234129 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.234101 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87"] Nov 03 02:30:01.302410 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.302376 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjmbc\" (UniqueName: \"kubernetes.io/projected/bbd12200-ef29-4e84-966d-5faed0aca233-kube-api-access-zjmbc\") pod \"maas-api-revoke-sub-30d3e735-qqw87\" (UID: \"bbd12200-ef29-4e84-966d-5faed0aca233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87" Nov 03 02:30:01.302538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.302460 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd12200-ef29-4e84-966d-5faed0aca233-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30d3e735-qqw87\" (UID: \"bbd12200-ef29-4e84-966d-5faed0aca233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87" Nov 03 02:30:01.302642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.302627 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd12200-ef29-4e84-966d-5faed0aca233-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30d3e735-qqw87\" (UID: \"bbd12200-ef29-4e84-966d-5faed0aca233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87" Nov 03 02:30:01.303115 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.303094 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd12200-ef29-4e84-966d-5faed0aca233-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30d3e735-qqw87\" (UID: \"bbd12200-ef29-4e84-966d-5faed0aca233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87" Nov 03 02:30:01.340136 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.340103 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl" event={"ID":"c0c965f8-4f67-420c-9322-0afa9e745187","Type":"ContainerStarted","Data":"ad082d0e94c99d2fc0c5367e1bcc602cf7eb197f6ffaa3c0cc35adcd634ba682"} Nov 03 02:30:01.341542 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.341511 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4" event={"ID":"c076195c-71f0-49cc-8aef-b690cab08678","Type":"ContainerStarted","Data":"4911c7b24f6e30bce3d4412acf1e031741f31ee34b11e62da759c9d5323835ec"} Nov 03 02:30:01.342902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.342880 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25" event={"ID":"e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f","Type":"ContainerStarted","Data":"90c269ad55fd1e2c03ee7802a8e3ef6eec10a338de7668e1da945eb964830143"} Nov 03 02:30:01.344605 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.344572 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" event={"ID":"cd0d24cd-c457-40b9-8786-378c54271667","Type":"ContainerStarted","Data":"ba8f439cbca0cc3b4bc10a244a32bab25be4487e0027a4e0aca2430f58599835"} Nov 03 02:30:01.344720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.344629 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" event={"ID":"cd0d24cd-c457-40b9-8786-378c54271667","Type":"ContainerStarted","Data":"f76d74b6a7856dfe2577c0858b2b507614c26f57d17c1211a3caf614bbd7a893"} Nov 03 02:30:01.345935 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.345901 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm" event={"ID":"0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98","Type":"ContainerStarted","Data":"cea1e2e735ff93909640ded046677edba57d8bcdcdb842a3ed63a81c76d7a168"} Nov 03 02:30:01.348139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.348112 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd12200-ef29-4e84-966d-5faed0aca233-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30d3e735-qqw87\" (UID: \"bbd12200-ef29-4e84-966d-5faed0aca233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87" Nov 03 02:30:01.360878 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.360857 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjmbc\" (UniqueName: \"kubernetes.io/projected/bbd12200-ef29-4e84-966d-5faed0aca233-kube-api-access-zjmbc\") pod \"maas-api-revoke-sub-30d3e735-qqw87\" (UID: \"bbd12200-ef29-4e84-966d-5faed0aca233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87" Nov 03 02:30:01.459507 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.459463 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" podStartSLOduration=1.459449478 podStartE2EDuration="1.459449478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:01.457862984 +0000 UTC m=+2283.141972140" watchObservedRunningTime="2026-11-03 02:30:01.459449478 +0000 UTC m=+2283.143558632" Nov 03 02:30:01.486587 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.486561 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87" Nov 03 02:30:01.861510 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:01.861473 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87"] Nov 03 02:30:02.361405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:02.361290 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87" event={"ID":"bbd12200-ef29-4e84-966d-5faed0aca233","Type":"ContainerStarted","Data":"1ff05024176cb82d2a928a45b3904114b3bdfe61a7d9856a1929cb9f2602ee87"} Nov 03 02:30:02.363275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:02.363249 5463 generic.go:356] "Generic (PLEG): container finished" podID="c0c965f8-4f67-420c-9322-0afa9e745187" containerID="6dc1e32fc1078e720160bb36332cbca5138970f828042f9cc4234f1bdce91aaa" exitCode=60 Nov 03 02:30:02.363408 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:02.363335 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl" event={"ID":"c0c965f8-4f67-420c-9322-0afa9e745187","Type":"ContainerDied","Data":"6dc1e32fc1078e720160bb36332cbca5138970f828042f9cc4234f1bdce91aaa"} Nov 03 02:30:02.363637 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:02.363614 5463 scope.go:122] "RemoveContainer" containerID="6dc1e32fc1078e720160bb36332cbca5138970f828042f9cc4234f1bdce91aaa" Nov 03 02:30:02.365404 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:02.365324 5463 generic.go:356] "Generic (PLEG): container finished" podID="c076195c-71f0-49cc-8aef-b690cab08678" containerID="b6095a821e8f4acf5020fccccd33b11937cf99e379ed6ee366ff087517989822" exitCode=60 Nov 03 02:30:02.365404 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:02.365389 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4" event={"ID":"c076195c-71f0-49cc-8aef-b690cab08678","Type":"ContainerDied","Data":"b6095a821e8f4acf5020fccccd33b11937cf99e379ed6ee366ff087517989822"} Nov 03 02:30:02.365661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:02.365644 5463 scope.go:122] "RemoveContainer" containerID="b6095a821e8f4acf5020fccccd33b11937cf99e379ed6ee366ff087517989822" Nov 03 02:30:02.370297 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:02.368496 5463 generic.go:356] "Generic (PLEG): container finished" podID="e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f" containerID="7a7a03b2414051ce9ff9e4b09a25c741b6c7378617c5937d8f5b64868168ad2f" exitCode=60 Nov 03 02:30:02.370297 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:02.368583 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25" event={"ID":"e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f","Type":"ContainerDied","Data":"7a7a03b2414051ce9ff9e4b09a25c741b6c7378617c5937d8f5b64868168ad2f"} Nov 03 02:30:02.370297 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:02.368817 5463 scope.go:122] "RemoveContainer" containerID="7a7a03b2414051ce9ff9e4b09a25c741b6c7378617c5937d8f5b64868168ad2f" Nov 03 02:30:02.370297 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:02.370179 5463 generic.go:356] "Generic (PLEG): container finished" podID="cd0d24cd-c457-40b9-8786-378c54271667" containerID="ba8f439cbca0cc3b4bc10a244a32bab25be4487e0027a4e0aca2430f58599835" exitCode=60 Nov 03 02:30:02.370297 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:02.370217 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" event={"ID":"cd0d24cd-c457-40b9-8786-378c54271667","Type":"ContainerDied","Data":"ba8f439cbca0cc3b4bc10a244a32bab25be4487e0027a4e0aca2430f58599835"} Nov 03 02:30:02.370570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:02.370449 5463 scope.go:122] "RemoveContainer" containerID="ba8f439cbca0cc3b4bc10a244a32bab25be4487e0027a4e0aca2430f58599835" Nov 03 02:30:02.371928 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:02.371905 5463 generic.go:356] "Generic (PLEG): container finished" podID="0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98" containerID="832e17cf4e8827340230bb4f7735919747fa0beeabc79d9f2c3cc57120fff54c" exitCode=60 Nov 03 02:30:02.372012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:02.371935 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm" event={"ID":"0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98","Type":"ContainerDied","Data":"832e17cf4e8827340230bb4f7735919747fa0beeabc79d9f2c3cc57120fff54c"} Nov 03 02:30:02.372124 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:02.372112 5463 scope.go:122] "RemoveContainer" containerID="832e17cf4e8827340230bb4f7735919747fa0beeabc79d9f2c3cc57120fff54c" Nov 03 02:30:03.385520 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:03.385493 5463 generic.go:356] "Generic (PLEG): container finished" podID="bbd12200-ef29-4e84-966d-5faed0aca233" containerID="30ac2cc6b30c8a6bff14c79e3162945128e60f455a45f7c78e1aba6d745d8406" exitCode=0 Nov 03 02:30:03.385932 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:03.385585 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87" event={"ID":"bbd12200-ef29-4e84-966d-5faed0aca233","Type":"ContainerDied","Data":"30ac2cc6b30c8a6bff14c79e3162945128e60f455a45f7c78e1aba6d745d8406"} Nov 03 02:30:03.387582 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:03.387561 5463 generic.go:356] "Generic (PLEG): container finished" podID="c0c965f8-4f67-420c-9322-0afa9e745187" containerID="8acd7ac315d1f79408f9697e1da92a057b6fafef93fae76ff55be0340a474eb0" exitCode=60 Nov 03 02:30:03.387704 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:03.387648 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl" event={"ID":"c0c965f8-4f67-420c-9322-0afa9e745187","Type":"ContainerDied","Data":"8acd7ac315d1f79408f9697e1da92a057b6fafef93fae76ff55be0340a474eb0"} Nov 03 02:30:03.387768 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:03.387710 5463 scope.go:122] "RemoveContainer" containerID="6dc1e32fc1078e720160bb36332cbca5138970f828042f9cc4234f1bdce91aaa" Nov 03 02:30:03.387931 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:03.387915 5463 scope.go:122] "RemoveContainer" containerID="8acd7ac315d1f79408f9697e1da92a057b6fafef93fae76ff55be0340a474eb0" Nov 03 02:30:03.388237 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:30:03.388219 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl_odh-ai-gateway-infra(c0c965f8-4f67-420c-9322-0afa9e745187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl" podUID="c0c965f8-4f67-420c-9322-0afa9e745187" Nov 03 02:30:03.390221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:03.390179 5463 generic.go:356] "Generic (PLEG): container finished" podID="c076195c-71f0-49cc-8aef-b690cab08678" containerID="126c2ab2d7804448742af2d01d4c6043fae1fba604069f0fa4d0dde88980e413" exitCode=60 Nov 03 02:30:03.390334 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:03.390239 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4" event={"ID":"c076195c-71f0-49cc-8aef-b690cab08678","Type":"ContainerDied","Data":"126c2ab2d7804448742af2d01d4c6043fae1fba604069f0fa4d0dde88980e413"} Nov 03 02:30:03.390502 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:03.390487 5463 scope.go:122] "RemoveContainer" containerID="126c2ab2d7804448742af2d01d4c6043fae1fba604069f0fa4d0dde88980e413" Nov 03 02:30:03.390806 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:30:03.390782 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894550-tgdk4_odh-ai-gateway-infra(c076195c-71f0-49cc-8aef-b690cab08678)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4" podUID="c076195c-71f0-49cc-8aef-b690cab08678" Nov 03 02:30:03.392185 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:03.392164 5463 generic.go:356] "Generic (PLEG): container finished" podID="e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f" containerID="cd27d0da96a547e3284adc43b0108098d98cf21fea09013c1b3b28ccd4491348" exitCode=60 Nov 03 02:30:03.392321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:03.392238 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25" event={"ID":"e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f","Type":"ContainerDied","Data":"cd27d0da96a547e3284adc43b0108098d98cf21fea09013c1b3b28ccd4491348"} Nov 03 02:30:03.392507 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:03.392490 5463 scope.go:122] "RemoveContainer" containerID="cd27d0da96a547e3284adc43b0108098d98cf21fea09013c1b3b28ccd4491348" Nov 03 02:30:03.392794 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:30:03.392771 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25_odh-ai-gateway-infra(e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25" podUID="e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f" Nov 03 02:30:03.394294 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:03.394273 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" event={"ID":"cd0d24cd-c457-40b9-8786-378c54271667","Type":"ContainerStarted","Data":"02c6a240e180bf825b1c5b89d7609a8c1c2b3455a7819325b08b86f7da65c1f7"} Nov 03 02:30:03.396631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:03.396613 5463 generic.go:356] "Generic (PLEG): container finished" podID="0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98" containerID="046c430ee6799a9f4d60c7e8c01ea7b870f2e649c6d0d9e9e96598ce9f97b838" exitCode=60 Nov 03 02:30:03.396722 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:03.396638 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm" event={"ID":"0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98","Type":"ContainerDied","Data":"046c430ee6799a9f4d60c7e8c01ea7b870f2e649c6d0d9e9e96598ce9f97b838"} Nov 03 02:30:03.396903 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:03.396887 5463 scope.go:122] "RemoveContainer" containerID="046c430ee6799a9f4d60c7e8c01ea7b870f2e649c6d0d9e9e96598ce9f97b838" Nov 03 02:30:03.397132 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:30:03.397110 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm_odh-ai-gateway-infra(0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm" podUID="0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98" Nov 03 02:30:03.655763 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:03.655737 5463 scope.go:122] "RemoveContainer" containerID="b6095a821e8f4acf5020fccccd33b11937cf99e379ed6ee366ff087517989822" Nov 03 02:30:03.669895 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:03.669872 5463 scope.go:122] "RemoveContainer" containerID="7a7a03b2414051ce9ff9e4b09a25c741b6c7378617c5937d8f5b64868168ad2f" Nov 03 02:30:03.683858 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:03.683779 5463 scope.go:122] "RemoveContainer" containerID="832e17cf4e8827340230bb4f7735919747fa0beeabc79d9f2c3cc57120fff54c" Nov 03 02:30:03.761430 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:03.761393 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:04.406852 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:04.406816 5463 generic.go:356] "Generic (PLEG): container finished" podID="cd0d24cd-c457-40b9-8786-378c54271667" containerID="02c6a240e180bf825b1c5b89d7609a8c1c2b3455a7819325b08b86f7da65c1f7" exitCode=60 Nov 03 02:30:04.407411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:04.406902 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" event={"ID":"cd0d24cd-c457-40b9-8786-378c54271667","Type":"ContainerDied","Data":"02c6a240e180bf825b1c5b89d7609a8c1c2b3455a7819325b08b86f7da65c1f7"} Nov 03 02:30:04.407411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:04.406965 5463 scope.go:122] "RemoveContainer" containerID="ba8f439cbca0cc3b4bc10a244a32bab25be4487e0027a4e0aca2430f58599835" Nov 03 02:30:04.407411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:04.407154 5463 scope.go:122] "RemoveContainer" containerID="02c6a240e180bf825b1c5b89d7609a8c1c2b3455a7819325b08b86f7da65c1f7" Nov 03 02:30:04.407583 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:30:04.407473 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222_odh-ai-gateway-infra(cd0d24cd-c457-40b9-8786-378c54271667)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" podUID="cd0d24cd-c457-40b9-8786-378c54271667" Nov 03 02:30:04.408981 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:04.408962 5463 scope.go:122] "RemoveContainer" containerID="046c430ee6799a9f4d60c7e8c01ea7b870f2e649c6d0d9e9e96598ce9f97b838" Nov 03 02:30:04.409252 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:30:04.409231 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm_odh-ai-gateway-infra(0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm" podUID="0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98" Nov 03 02:30:04.410761 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:04.410741 5463 scope.go:122] "RemoveContainer" containerID="8acd7ac315d1f79408f9697e1da92a057b6fafef93fae76ff55be0340a474eb0" Nov 03 02:30:04.410980 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:30:04.410960 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl_odh-ai-gateway-infra(c0c965f8-4f67-420c-9322-0afa9e745187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl" podUID="c0c965f8-4f67-420c-9322-0afa9e745187" Nov 03 02:30:04.412471 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:04.412452 5463 scope.go:122] "RemoveContainer" containerID="126c2ab2d7804448742af2d01d4c6043fae1fba604069f0fa4d0dde88980e413" Nov 03 02:30:04.412720 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:30:04.412693 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894550-tgdk4_odh-ai-gateway-infra(c076195c-71f0-49cc-8aef-b690cab08678)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4" podUID="c076195c-71f0-49cc-8aef-b690cab08678" Nov 03 02:30:04.414068 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:04.414049 5463 scope.go:122] "RemoveContainer" containerID="cd27d0da96a547e3284adc43b0108098d98cf21fea09013c1b3b28ccd4491348" Nov 03 02:30:04.414340 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:30:04.414318 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25_odh-ai-gateway-infra(e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25" podUID="e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f" Nov 03 02:30:04.569796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:04.569762 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87" Nov 03 02:30:04.645507 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:04.645483 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd12200-ef29-4e84-966d-5faed0aca233-kube-api-access-zjmbc\" (UniqueName: \"kubernetes.io/projected/bbd12200-ef29-4e84-966d-5faed0aca233-kube-api-access-zjmbc\") pod \"bbd12200-ef29-4e84-966d-5faed0aca233\" (UID: \"bbd12200-ef29-4e84-966d-5faed0aca233\") " Nov 03 02:30:04.645593 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:04.645538 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbd12200-ef29-4e84-966d-5faed0aca233-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd12200-ef29-4e84-966d-5faed0aca233-maas-api-openshift-ca\") pod \"bbd12200-ef29-4e84-966d-5faed0aca233\" (UID: \"bbd12200-ef29-4e84-966d-5faed0aca233\") " Nov 03 02:30:04.645638 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:04.645598 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbd12200-ef29-4e84-966d-5faed0aca233-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd12200-ef29-4e84-966d-5faed0aca233-maas-api-xks-ca\") pod \"bbd12200-ef29-4e84-966d-5faed0aca233\" (UID: \"bbd12200-ef29-4e84-966d-5faed0aca233\") " Nov 03 02:30:04.645965 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:04.645935 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd12200-ef29-4e84-966d-5faed0aca233-maas-api-openshift-ca" pod "bbd12200-ef29-4e84-966d-5faed0aca233" (UID: "bbd12200-ef29-4e84-966d-5faed0aca233"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:04.646179 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:04.646163 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd12200-ef29-4e84-966d-5faed0aca233-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:04.647865 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:04.647839 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbd12200-ef29-4e84-966d-5faed0aca233-maas-api-xks-ca" pod "bbd12200-ef29-4e84-966d-5faed0aca233" (UID: "bbd12200-ef29-4e84-966d-5faed0aca233"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:30:04.647957 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:04.647862 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd12200-ef29-4e84-966d-5faed0aca233-kube-api-access-zjmbc" pod "bbd12200-ef29-4e84-966d-5faed0aca233" (UID: "bbd12200-ef29-4e84-966d-5faed0aca233"). InnerVolumeSpecName "kube-api-access-zjmbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:04.746913 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:04.746886 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjmbc\" (UniqueName: \"kubernetes.io/projected/bbd12200-ef29-4e84-966d-5faed0aca233-kube-api-access-zjmbc\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:04.746913 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:04.746913 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd12200-ef29-4e84-966d-5faed0aca233-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:05.424081 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:05.424055 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87" Nov 03 02:30:05.424499 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:05.424087 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87" event={"ID":"bbd12200-ef29-4e84-966d-5faed0aca233","Type":"ContainerDied","Data":"1ff05024176cb82d2a928a45b3904114b3bdfe61a7d9856a1929cb9f2602ee87"} Nov 03 02:30:05.424499 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:05.424135 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ff05024176cb82d2a928a45b3904114b3bdfe61a7d9856a1929cb9f2602ee87" Nov 03 02:30:05.426504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:05.426477 5463 scope.go:122] "RemoveContainer" containerID="02c6a240e180bf825b1c5b89d7609a8c1c2b3455a7819325b08b86f7da65c1f7" Nov 03 02:30:05.426786 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:30:05.426765 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222_odh-ai-gateway-infra(cd0d24cd-c457-40b9-8786-378c54271667)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" podUID="cd0d24cd-c457-40b9-8786-378c54271667" Nov 03 02:30:08.510373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:08.510330 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:12.629081 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:12.629038 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:12.681012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:12.680967 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm"] Nov 03 02:30:12.852560 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:12.852533 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm" Nov 03 02:30:12.943707 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:12.943678 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98-kube-api-access-zk2vf\" (UniqueName: \"kubernetes.io/projected/0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98-kube-api-access-zk2vf\") pod \"0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98\" (UID: \"0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98\") " Nov 03 02:30:12.943892 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:12.943868 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98-maas-api-service-ca\") pod \"0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98\" (UID: \"0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98\") " Nov 03 02:30:12.944776 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:12.944752 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98-maas-api-service-ca" pod "0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98" (UID: "0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:12.946457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:12.946428 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98-kube-api-access-zk2vf" pod "0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98" (UID: "0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98"). InnerVolumeSpecName "kube-api-access-zk2vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:12.952587 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:12.952557 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw"] Nov 03 02:30:12.952796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:12.952768 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw" podUID="9c92a303-5b7f-438e-8839-6ae9148d340a" containerName="payload-processing" containerID="cri-o://4068599889c8f29b131d5b7da710cb3af3d5409ea8f8ac4bae6cfa3779677a2d" gracePeriod=30 Nov 03 02:30:13.006038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.006011 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9"] Nov 03 02:30:13.006263 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.006229 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9" podUID="c97f240a-07b4-4e32-be49-4b7a9f077012" containerName="payload-pre-processing" containerID="cri-o://32ad3e16c2a4c349a6c30cae5d920bd2678305a20ce5a4a956a4964777188c73" gracePeriod=30 Nov 03 02:30:13.025119 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:30:13.025085 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:30:13.045736 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.045709 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:13.045876 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.045740 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk2vf\" (UniqueName: \"kubernetes.io/projected/0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98-kube-api-access-zk2vf\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:13.120769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.120684 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:13.261152 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.261111 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:13.296237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.296209 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw" Nov 03 02:30:13.314103 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.314081 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9" Nov 03 02:30:13.350842 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.350816 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c97f240a-07b4-4e32-be49-4b7a9f077012-kube-api-access-tlb5b\" (UniqueName: \"kubernetes.io/projected/c97f240a-07b4-4e32-be49-4b7a9f077012-kube-api-access-tlb5b\") pod \"c97f240a-07b4-4e32-be49-4b7a9f077012\" (UID: \"c97f240a-07b4-4e32-be49-4b7a9f077012\") " Nov 03 02:30:13.350977 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.350878 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c92a303-5b7f-438e-8839-6ae9148d340a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c92a303-5b7f-438e-8839-6ae9148d340a-plugins-config-volume\") pod \"9c92a303-5b7f-438e-8839-6ae9148d340a\" (UID: \"9c92a303-5b7f-438e-8839-6ae9148d340a\") " Nov 03 02:30:13.351111 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.351032 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c92a303-5b7f-438e-8839-6ae9148d340a-kube-api-access-bnlq7\" (UniqueName: \"kubernetes.io/projected/9c92a303-5b7f-438e-8839-6ae9148d340a-kube-api-access-bnlq7\") pod \"9c92a303-5b7f-438e-8839-6ae9148d340a\" (UID: \"9c92a303-5b7f-438e-8839-6ae9148d340a\") " Nov 03 02:30:13.351111 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.351078 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c97f240a-07b4-4e32-be49-4b7a9f077012-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97f240a-07b4-4e32-be49-4b7a9f077012-plugins-config-volume\") pod \"c97f240a-07b4-4e32-be49-4b7a9f077012\" (UID: \"c97f240a-07b4-4e32-be49-4b7a9f077012\") " Nov 03 02:30:13.351323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.351301 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c92a303-5b7f-438e-8839-6ae9148d340a-plugins-config-volume" pod "9c92a303-5b7f-438e-8839-6ae9148d340a" (UID: "9c92a303-5b7f-438e-8839-6ae9148d340a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:13.351561 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.351535 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c97f240a-07b4-4e32-be49-4b7a9f077012-plugins-config-volume" pod "c97f240a-07b4-4e32-be49-4b7a9f077012" (UID: "c97f240a-07b4-4e32-be49-4b7a9f077012"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:13.351699 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.351679 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97f240a-07b4-4e32-be49-4b7a9f077012-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:13.351766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.351703 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c92a303-5b7f-438e-8839-6ae9148d340a-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:13.353018 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.352991 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97f240a-07b4-4e32-be49-4b7a9f077012-kube-api-access-tlb5b" pod "c97f240a-07b4-4e32-be49-4b7a9f077012" (UID: "c97f240a-07b4-4e32-be49-4b7a9f077012"). InnerVolumeSpecName "kube-api-access-tlb5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:13.353234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.353183 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c92a303-5b7f-438e-8839-6ae9148d340a-kube-api-access-bnlq7" pod "9c92a303-5b7f-438e-8839-6ae9148d340a" (UID: "9c92a303-5b7f-438e-8839-6ae9148d340a"). InnerVolumeSpecName "kube-api-access-bnlq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:13.454270 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.453256 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlb5b\" (UniqueName: \"kubernetes.io/projected/c97f240a-07b4-4e32-be49-4b7a9f077012-kube-api-access-tlb5b\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:13.454270 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.453289 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnlq7\" (UniqueName: \"kubernetes.io/projected/9c92a303-5b7f-438e-8839-6ae9148d340a-kube-api-access-bnlq7\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:13.552219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.545791 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm" Nov 03 02:30:13.552219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.546489 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm" event={"ID":"0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98","Type":"ContainerDied","Data":"cea1e2e735ff93909640ded046677edba57d8bcdcdb842a3ed63a81c76d7a168"} Nov 03 02:30:13.552219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.546581 5463 scope.go:122] "RemoveContainer" containerID="046c430ee6799a9f4d60c7e8c01ea7b870f2e649c6d0d9e9e96598ce9f97b838" Nov 03 02:30:13.552219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.551243 5463 generic.go:356] "Generic (PLEG): container finished" podID="c97f240a-07b4-4e32-be49-4b7a9f077012" containerID="32ad3e16c2a4c349a6c30cae5d920bd2678305a20ce5a4a956a4964777188c73" exitCode=0 Nov 03 02:30:13.552219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.551401 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9" event={"ID":"c97f240a-07b4-4e32-be49-4b7a9f077012","Type":"ContainerDied","Data":"32ad3e16c2a4c349a6c30cae5d920bd2678305a20ce5a4a956a4964777188c73"} Nov 03 02:30:13.552219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.551446 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9" event={"ID":"c97f240a-07b4-4e32-be49-4b7a9f077012","Type":"ContainerDied","Data":"b49229f7455d7c132291c423b45ef188ced27cdd5afe2f2fe78ec4a0bd449b79"} Nov 03 02:30:13.552219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.551524 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9" Nov 03 02:30:13.555223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.554052 5463 generic.go:356] "Generic (PLEG): container finished" podID="9c92a303-5b7f-438e-8839-6ae9148d340a" containerID="4068599889c8f29b131d5b7da710cb3af3d5409ea8f8ac4bae6cfa3779677a2d" exitCode=0 Nov 03 02:30:13.555223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.554092 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw" event={"ID":"9c92a303-5b7f-438e-8839-6ae9148d340a","Type":"ContainerDied","Data":"4068599889c8f29b131d5b7da710cb3af3d5409ea8f8ac4bae6cfa3779677a2d"} Nov 03 02:30:13.555223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.554130 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw" event={"ID":"9c92a303-5b7f-438e-8839-6ae9148d340a","Type":"ContainerDied","Data":"fe67ce2b15b00eab8d924d5f2843e9f91592b0208e6b4efa9a7c0717c9d3f3ff"} Nov 03 02:30:13.555223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.554221 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw" Nov 03 02:30:13.576712 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.574512 5463 scope.go:122] "RemoveContainer" containerID="32ad3e16c2a4c349a6c30cae5d920bd2678305a20ce5a4a956a4964777188c73" Nov 03 02:30:13.590694 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.590676 5463 scope.go:122] "RemoveContainer" containerID="32ad3e16c2a4c349a6c30cae5d920bd2678305a20ce5a4a956a4964777188c73" Nov 03 02:30:13.591370 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:30:13.591163 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ad3e16c2a4c349a6c30cae5d920bd2678305a20ce5a4a956a4964777188c73\": container with ID starting with 32ad3e16c2a4c349a6c30cae5d920bd2678305a20ce5a4a956a4964777188c73 not found: ID does not exist" containerID="32ad3e16c2a4c349a6c30cae5d920bd2678305a20ce5a4a956a4964777188c73" Nov 03 02:30:13.591370 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.591245 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ad3e16c2a4c349a6c30cae5d920bd2678305a20ce5a4a956a4964777188c73"} err="failed to get container status \"32ad3e16c2a4c349a6c30cae5d920bd2678305a20ce5a4a956a4964777188c73\": rpc error: code = NotFound desc = could not find container \"32ad3e16c2a4c349a6c30cae5d920bd2678305a20ce5a4a956a4964777188c73\": container with ID starting with 32ad3e16c2a4c349a6c30cae5d920bd2678305a20ce5a4a956a4964777188c73 not found: ID does not exist" Nov 03 02:30:13.591370 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.591272 5463 scope.go:122] "RemoveContainer" containerID="4068599889c8f29b131d5b7da710cb3af3d5409ea8f8ac4bae6cfa3779677a2d" Nov 03 02:30:13.603907 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.603865 5463 scope.go:122] "RemoveContainer" containerID="4068599889c8f29b131d5b7da710cb3af3d5409ea8f8ac4bae6cfa3779677a2d" Nov 03 02:30:13.604139 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:30:13.604121 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4068599889c8f29b131d5b7da710cb3af3d5409ea8f8ac4bae6cfa3779677a2d\": container with ID starting with 4068599889c8f29b131d5b7da710cb3af3d5409ea8f8ac4bae6cfa3779677a2d not found: ID does not exist" containerID="4068599889c8f29b131d5b7da710cb3af3d5409ea8f8ac4bae6cfa3779677a2d" Nov 03 02:30:13.604270 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.604166 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4068599889c8f29b131d5b7da710cb3af3d5409ea8f8ac4bae6cfa3779677a2d"} err="failed to get container status \"4068599889c8f29b131d5b7da710cb3af3d5409ea8f8ac4bae6cfa3779677a2d\": rpc error: code = NotFound desc = could not find container \"4068599889c8f29b131d5b7da710cb3af3d5409ea8f8ac4bae6cfa3779677a2d\": container with ID starting with 4068599889c8f29b131d5b7da710cb3af3d5409ea8f8ac4bae6cfa3779677a2d not found: ID does not exist" Nov 03 02:30:13.844225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.844181 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw"] Nov 03 02:30:13.847806 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.847778 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-56299c-557f8b65d6-8g6dw"] Nov 03 02:30:13.957642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.957587 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm"] Nov 03 02:30:13.960788 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:13.960765 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-56299c-29894550-gv8xm"] Nov 03 02:30:14.072278 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:14.072250 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9"] Nov 03 02:30:14.075146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:14.075124 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-56299c-6979fcff6c-k47f9"] Nov 03 02:30:14.436421 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:14.436389 5463 scope.go:122] "RemoveContainer" containerID="02c6a240e180bf825b1c5b89d7609a8c1c2b3455a7819325b08b86f7da65c1f7" Nov 03 02:30:14.451975 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:14.451943 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98" path="/var/lib/kubelet/pods/0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98/volumes" Nov 03 02:30:14.452674 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:14.452648 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c92a303-5b7f-438e-8839-6ae9148d340a" path="/var/lib/kubelet/pods/9c92a303-5b7f-438e-8839-6ae9148d340a/volumes" Nov 03 02:30:14.453309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:14.453283 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c97f240a-07b4-4e32-be49-4b7a9f077012" path="/var/lib/kubelet/pods/c97f240a-07b4-4e32-be49-4b7a9f077012/volumes" Nov 03 02:30:14.521835 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:14.521789 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:15.580119 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:15.580066 5463 generic.go:356] "Generic (PLEG): container finished" podID="cd0d24cd-c457-40b9-8786-378c54271667" containerID="fe6ac3720bd5326a3274361fca90da9a52ca17c99883d5882c2114d70c72f782" exitCode=60 Nov 03 02:30:15.580615 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:15.580137 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" event={"ID":"cd0d24cd-c457-40b9-8786-378c54271667","Type":"ContainerDied","Data":"fe6ac3720bd5326a3274361fca90da9a52ca17c99883d5882c2114d70c72f782"} Nov 03 02:30:15.580615 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:15.580227 5463 scope.go:122] "RemoveContainer" containerID="02c6a240e180bf825b1c5b89d7609a8c1c2b3455a7819325b08b86f7da65c1f7" Nov 03 02:30:15.580785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:15.580684 5463 scope.go:122] "RemoveContainer" containerID="fe6ac3720bd5326a3274361fca90da9a52ca17c99883d5882c2114d70c72f782" Nov 03 02:30:15.580988 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:30:15.580963 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222_odh-ai-gateway-infra(cd0d24cd-c457-40b9-8786-378c54271667)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" podUID="cd0d24cd-c457-40b9-8786-378c54271667" Nov 03 02:30:15.940179 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:15.940144 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:16.435962 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:16.435934 5463 scope.go:122] "RemoveContainer" containerID="126c2ab2d7804448742af2d01d4c6043fae1fba604069f0fa4d0dde88980e413" Nov 03 02:30:16.620793 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:16.620761 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:16.734376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:16.734275 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222"] Nov 03 02:30:16.999693 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:16.999626 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" Nov 03 02:30:17.014234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.014183 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0d24cd-c457-40b9-8786-378c54271667-kube-api-access-7rr9r\" (UniqueName: \"kubernetes.io/projected/cd0d24cd-c457-40b9-8786-378c54271667-kube-api-access-7rr9r\") pod \"cd0d24cd-c457-40b9-8786-378c54271667\" (UID: \"cd0d24cd-c457-40b9-8786-378c54271667\") " Nov 03 02:30:17.014365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.014340 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0d24cd-c457-40b9-8786-378c54271667-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd0d24cd-c457-40b9-8786-378c54271667-maas-api-service-ca\") pod \"cd0d24cd-c457-40b9-8786-378c54271667\" (UID: \"cd0d24cd-c457-40b9-8786-378c54271667\") " Nov 03 02:30:17.014713 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.014687 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0d24cd-c457-40b9-8786-378c54271667-maas-api-service-ca" pod "cd0d24cd-c457-40b9-8786-378c54271667" (UID: "cd0d24cd-c457-40b9-8786-378c54271667"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:17.014855 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.014838 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd0d24cd-c457-40b9-8786-378c54271667-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:17.016282 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.016258 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0d24cd-c457-40b9-8786-378c54271667-kube-api-access-7rr9r" pod "cd0d24cd-c457-40b9-8786-378c54271667" (UID: "cd0d24cd-c457-40b9-8786-378c54271667"). InnerVolumeSpecName "kube-api-access-7rr9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:17.116047 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.116016 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rr9r\" (UniqueName: \"kubernetes.io/projected/cd0d24cd-c457-40b9-8786-378c54271667-kube-api-access-7rr9r\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:17.237456 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.237417 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw"] Nov 03 02:30:17.238080 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.238064 5463 cpu_manager.go:414] "containerMap: removing container" podUID="cd0d24cd-c457-40b9-8786-378c54271667" containerName="cleanup" Nov 03 02:30:17.238129 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.238081 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0d24cd-c457-40b9-8786-378c54271667" containerName="cleanup" Nov 03 02:30:17.238129 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.238092 5463 cpu_manager.go:414] "containerMap: removing container" podUID="cd0d24cd-c457-40b9-8786-378c54271667" containerName="cleanup" Nov 03 02:30:17.238129 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.238098 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0d24cd-c457-40b9-8786-378c54271667" containerName="cleanup" Nov 03 02:30:17.238129 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.238125 5463 cpu_manager.go:414] "containerMap: removing container" podUID="9c92a303-5b7f-438e-8839-6ae9148d340a" containerName="payload-processing" Nov 03 02:30:17.238129 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.238129 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c92a303-5b7f-438e-8839-6ae9148d340a" containerName="payload-processing" Nov 03 02:30:17.238306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.238148 5463 cpu_manager.go:414] "containerMap: removing container" podUID="c97f240a-07b4-4e32-be49-4b7a9f077012" containerName="payload-pre-processing" Nov 03 02:30:17.238306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.238153 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c97f240a-07b4-4e32-be49-4b7a9f077012" containerName="payload-pre-processing" Nov 03 02:30:17.238306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.238164 5463 cpu_manager.go:414] "containerMap: removing container" podUID="0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98" containerName="cleanup" Nov 03 02:30:17.238306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.238169 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98" containerName="cleanup" Nov 03 02:30:17.238306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.238183 5463 cpu_manager.go:414] "containerMap: removing container" podUID="bbd12200-ef29-4e84-966d-5faed0aca233" containerName="revoke-keys" Nov 03 02:30:17.238306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.238202 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd12200-ef29-4e84-966d-5faed0aca233" containerName="revoke-keys" Nov 03 02:30:17.238306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.238300 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c92a303-5b7f-438e-8839-6ae9148d340a" containerName="payload-processing" Nov 03 02:30:17.238525 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.238311 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c97f240a-07b4-4e32-be49-4b7a9f077012" containerName="payload-pre-processing" Nov 03 02:30:17.238525 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.238318 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0d24cd-c457-40b9-8786-378c54271667" containerName="cleanup" Nov 03 02:30:17.238525 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.238323 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98" containerName="cleanup" Nov 03 02:30:17.238525 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.238332 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98" containerName="cleanup" Nov 03 02:30:17.238525 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.238338 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd12200-ef29-4e84-966d-5faed0aca233" containerName="revoke-keys" Nov 03 02:30:17.238525 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.238346 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0d24cd-c457-40b9-8786-378c54271667" containerName="cleanup" Nov 03 02:30:17.238525 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.238352 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0d24cd-c457-40b9-8786-378c54271667" containerName="cleanup" Nov 03 02:30:17.241908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.241889 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw" Nov 03 02:30:17.294329 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.294258 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw"] Nov 03 02:30:17.318766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.318740 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rgdd\" (UniqueName: \"kubernetes.io/projected/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-kube-api-access-2rgdd\") pod \"maas-api-revoke-sub-7883bf4a-xlmtw\" (UID: \"a8ba3dae-180f-4d19-a5d3-fb4dbf253295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw" Nov 03 02:30:17.318927 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.318789 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7883bf4a-xlmtw\" (UID: \"a8ba3dae-180f-4d19-a5d3-fb4dbf253295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw" Nov 03 02:30:17.318927 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.318897 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7883bf4a-xlmtw\" (UID: \"a8ba3dae-180f-4d19-a5d3-fb4dbf253295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw" Nov 03 02:30:17.374305 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.374272 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:30:17.420018 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.419978 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rgdd\" (UniqueName: \"kubernetes.io/projected/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-kube-api-access-2rgdd\") pod \"maas-api-revoke-sub-7883bf4a-xlmtw\" (UID: \"a8ba3dae-180f-4d19-a5d3-fb4dbf253295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw" Nov 03 02:30:17.420254 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.420033 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7883bf4a-xlmtw\" (UID: \"a8ba3dae-180f-4d19-a5d3-fb4dbf253295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw" Nov 03 02:30:17.420254 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.420077 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7883bf4a-xlmtw\" (UID: \"a8ba3dae-180f-4d19-a5d3-fb4dbf253295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw" Nov 03 02:30:17.421003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.420976 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7883bf4a-xlmtw\" (UID: \"a8ba3dae-180f-4d19-a5d3-fb4dbf253295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw" Nov 03 02:30:17.422587 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.422558 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7883bf4a-xlmtw\" (UID: \"a8ba3dae-180f-4d19-a5d3-fb4dbf253295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw" Nov 03 02:30:17.435825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.435794 5463 scope.go:122] "RemoveContainer" containerID="cd27d0da96a547e3284adc43b0108098d98cf21fea09013c1b3b28ccd4491348" Nov 03 02:30:17.477681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.477617 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rgdd\" (UniqueName: \"kubernetes.io/projected/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-kube-api-access-2rgdd\") pod \"maas-api-revoke-sub-7883bf4a-xlmtw\" (UID: \"a8ba3dae-180f-4d19-a5d3-fb4dbf253295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw" Nov 03 02:30:17.558737 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.558710 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw" Nov 03 02:30:17.614385 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.614344 5463 generic.go:356] "Generic (PLEG): container finished" podID="c076195c-71f0-49cc-8aef-b690cab08678" containerID="3859d4bd42f141435ed3635c556aed689b1d530096d1319af2767a8e0bf33bc0" exitCode=60 Nov 03 02:30:17.614564 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.614528 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4" event={"ID":"c076195c-71f0-49cc-8aef-b690cab08678","Type":"ContainerDied","Data":"3859d4bd42f141435ed3635c556aed689b1d530096d1319af2767a8e0bf33bc0"} Nov 03 02:30:17.614687 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.614590 5463 scope.go:122] "RemoveContainer" containerID="126c2ab2d7804448742af2d01d4c6043fae1fba604069f0fa4d0dde88980e413" Nov 03 02:30:17.615258 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.615034 5463 scope.go:122] "RemoveContainer" containerID="3859d4bd42f141435ed3635c556aed689b1d530096d1319af2767a8e0bf33bc0" Nov 03 02:30:17.615443 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:30:17.615415 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894550-tgdk4_odh-ai-gateway-infra(c076195c-71f0-49cc-8aef-b690cab08678)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4" podUID="c076195c-71f0-49cc-8aef-b690cab08678" Nov 03 02:30:17.618776 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.618749 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" Nov 03 02:30:17.618872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.618845 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222" event={"ID":"cd0d24cd-c457-40b9-8786-378c54271667","Type":"ContainerDied","Data":"f76d74b6a7856dfe2577c0858b2b507614c26f57d17c1211a3caf614bbd7a893"} Nov 03 02:30:17.635482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.635459 5463 scope.go:122] "RemoveContainer" containerID="fe6ac3720bd5326a3274361fca90da9a52ca17c99883d5882c2114d70c72f782" Nov 03 02:30:17.842881 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.842810 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222"] Nov 03 02:30:17.866854 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.847745 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7f760c-29894550-kz222"] Nov 03 02:30:17.922310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.922279 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:17.977147 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:30:17.977113 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ba3dae_180f_4d19_a5d3_fb4dbf253295.slice/crio-ecff6a397defd376919ea36c8c5756ada76c722c6e8c3bdce9867f222e59e28e WatchSource:0}: Error finding container ecff6a397defd376919ea36c8c5756ada76c722c6e8c3bdce9867f222e59e28e: Status 404 returned error can't find the container with id ecff6a397defd376919ea36c8c5756ada76c722c6e8c3bdce9867f222e59e28e Nov 03 02:30:17.978473 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:17.978445 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw"] Nov 03 02:30:18.074158 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:18.074115 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:18.439574 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:18.439511 5463 scope.go:122] "RemoveContainer" containerID="8acd7ac315d1f79408f9697e1da92a057b6fafef93fae76ff55be0340a474eb0" Nov 03 02:30:18.445547 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:18.445518 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd0d24cd-c457-40b9-8786-378c54271667" path="/var/lib/kubelet/pods/cd0d24cd-c457-40b9-8786-378c54271667/volumes" Nov 03 02:30:18.630916 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:18.630881 5463 generic.go:356] "Generic (PLEG): container finished" podID="e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f" containerID="a80b17dd057a5edde0e3b23aae1f04ae6624bd3a1232114c3019025d7cb74dd3" exitCode=60 Nov 03 02:30:18.631145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:18.630959 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25" event={"ID":"e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f","Type":"ContainerDied","Data":"a80b17dd057a5edde0e3b23aae1f04ae6624bd3a1232114c3019025d7cb74dd3"} Nov 03 02:30:18.631145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:18.631063 5463 scope.go:122] "RemoveContainer" containerID="cd27d0da96a547e3284adc43b0108098d98cf21fea09013c1b3b28ccd4491348" Nov 03 02:30:18.631514 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:18.631498 5463 scope.go:122] "RemoveContainer" containerID="a80b17dd057a5edde0e3b23aae1f04ae6624bd3a1232114c3019025d7cb74dd3" Nov 03 02:30:18.631839 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:30:18.631818 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25_odh-ai-gateway-infra(e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25" podUID="e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f" Nov 03 02:30:18.633729 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:18.633703 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw" event={"ID":"a8ba3dae-180f-4d19-a5d3-fb4dbf253295","Type":"ContainerStarted","Data":"dc7a4fa162e005dcf75bb8856db2c837e987e0ff24be5744522e8187a5083981"} Nov 03 02:30:18.633816 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:18.633752 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw" event={"ID":"a8ba3dae-180f-4d19-a5d3-fb4dbf253295","Type":"ContainerStarted","Data":"ecff6a397defd376919ea36c8c5756ada76c722c6e8c3bdce9867f222e59e28e"} Nov 03 02:30:18.738720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:18.738647 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4"] Nov 03 02:30:18.745318 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:18.745280 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:18.862149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:18.862088 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw" podStartSLOduration=1.8620715030000001 podStartE2EDuration="1.862071503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:18.859995077 +0000 UTC m=+2300.544104234" watchObservedRunningTime="2026-11-03 02:30:18.862071503 +0000 UTC m=+2300.546180661" Nov 03 02:30:18.888439 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:18.888415 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4" Nov 03 02:30:18.938355 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:18.938325 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c076195c-71f0-49cc-8aef-b690cab08678-kube-api-access-d4kkl\" (UniqueName: \"kubernetes.io/projected/c076195c-71f0-49cc-8aef-b690cab08678-kube-api-access-d4kkl\") pod \"c076195c-71f0-49cc-8aef-b690cab08678\" (UID: \"c076195c-71f0-49cc-8aef-b690cab08678\") " Nov 03 02:30:18.938479 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:18.938374 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c076195c-71f0-49cc-8aef-b690cab08678-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c076195c-71f0-49cc-8aef-b690cab08678-maas-api-service-ca\") pod \"c076195c-71f0-49cc-8aef-b690cab08678\" (UID: \"c076195c-71f0-49cc-8aef-b690cab08678\") " Nov 03 02:30:18.938779 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:18.938759 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c076195c-71f0-49cc-8aef-b690cab08678-maas-api-service-ca" pod "c076195c-71f0-49cc-8aef-b690cab08678" (UID: "c076195c-71f0-49cc-8aef-b690cab08678"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:18.940381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:18.940362 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c076195c-71f0-49cc-8aef-b690cab08678-kube-api-access-d4kkl" pod "c076195c-71f0-49cc-8aef-b690cab08678" (UID: "c076195c-71f0-49cc-8aef-b690cab08678"). InnerVolumeSpecName "kube-api-access-d4kkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:19.039825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:19.039799 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4kkl\" (UniqueName: \"kubernetes.io/projected/c076195c-71f0-49cc-8aef-b690cab08678-kube-api-access-d4kkl\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:19.039825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:19.039828 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c076195c-71f0-49cc-8aef-b690cab08678-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:19.642074 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:19.641993 5463 generic.go:356] "Generic (PLEG): container finished" podID="c0c965f8-4f67-420c-9322-0afa9e745187" containerID="ccb5fe9c862cf5d9f58dd6be49f1289f5cdebb62b0331fcd28e437a2f4d31691" exitCode=60 Nov 03 02:30:19.642274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:19.642066 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl" event={"ID":"c0c965f8-4f67-420c-9322-0afa9e745187","Type":"ContainerDied","Data":"ccb5fe9c862cf5d9f58dd6be49f1289f5cdebb62b0331fcd28e437a2f4d31691"} Nov 03 02:30:19.642274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:19.642133 5463 scope.go:122] "RemoveContainer" containerID="8acd7ac315d1f79408f9697e1da92a057b6fafef93fae76ff55be0340a474eb0" Nov 03 02:30:19.642544 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:19.642526 5463 scope.go:122] "RemoveContainer" containerID="ccb5fe9c862cf5d9f58dd6be49f1289f5cdebb62b0331fcd28e437a2f4d31691" Nov 03 02:30:19.642857 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:30:19.642838 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl_odh-ai-gateway-infra(c0c965f8-4f67-420c-9322-0afa9e745187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl" podUID="c0c965f8-4f67-420c-9322-0afa9e745187" Nov 03 02:30:19.643473 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:19.643459 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4" Nov 03 02:30:19.643546 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:19.643477 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4" event={"ID":"c076195c-71f0-49cc-8aef-b690cab08678","Type":"ContainerDied","Data":"4911c7b24f6e30bce3d4412acf1e031741f31ee34b11e62da759c9d5323835ec"} Nov 03 02:30:19.650182 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:19.650153 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:19.652737 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:19.652712 5463 generic.go:356] "Generic (PLEG): container finished" podID="a8ba3dae-180f-4d19-a5d3-fb4dbf253295" containerID="dc7a4fa162e005dcf75bb8856db2c837e987e0ff24be5744522e8187a5083981" exitCode=0 Nov 03 02:30:19.652864 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:19.652796 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw" event={"ID":"a8ba3dae-180f-4d19-a5d3-fb4dbf253295","Type":"ContainerDied","Data":"dc7a4fa162e005dcf75bb8856db2c837e987e0ff24be5744522e8187a5083981"} Nov 03 02:30:19.658995 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:19.658977 5463 scope.go:122] "RemoveContainer" containerID="3859d4bd42f141435ed3635c556aed689b1d530096d1319af2767a8e0bf33bc0" Nov 03 02:30:19.774794 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:19.774763 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25"] Nov 03 02:30:19.910333 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:19.910285 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:19.976614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:19.976578 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25" Nov 03 02:30:20.055611 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.055574 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f-kube-api-access-wm2sw\" (UniqueName: \"kubernetes.io/projected/e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f-kube-api-access-wm2sw\") pod \"e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f\" (UID: \"e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f\") " Nov 03 02:30:20.055849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.055697 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f-maas-api-service-ca\") pod \"e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f\" (UID: \"e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f\") " Nov 03 02:30:20.056476 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.056445 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f-maas-api-service-ca" pod "e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f" (UID: "e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:20.063359 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.063327 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f-kube-api-access-wm2sw" pod "e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f" (UID: "e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f"). InnerVolumeSpecName "kube-api-access-wm2sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:20.073347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.073319 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:20.139917 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.139888 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4"] Nov 03 02:30:20.143152 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.143131 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tgdk4"] Nov 03 02:30:20.160492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.160373 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:20.160492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.160410 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm2sw\" (UniqueName: \"kubernetes.io/projected/e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f-kube-api-access-wm2sw\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:20.407707 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.407678 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d"] Nov 03 02:30:20.408358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.408344 5463 cpu_manager.go:414] "containerMap: removing container" podUID="c076195c-71f0-49cc-8aef-b690cab08678" containerName="cleanup" Nov 03 02:30:20.408401 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.408359 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c076195c-71f0-49cc-8aef-b690cab08678" containerName="cleanup" Nov 03 02:30:20.408401 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.408382 5463 cpu_manager.go:414] "containerMap: removing container" podUID="cd0d24cd-c457-40b9-8786-378c54271667" containerName="cleanup" Nov 03 02:30:20.408401 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.408387 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0d24cd-c457-40b9-8786-378c54271667" containerName="cleanup" Nov 03 02:30:20.408401 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.408394 5463 cpu_manager.go:414] "containerMap: removing container" podUID="c076195c-71f0-49cc-8aef-b690cab08678" containerName="cleanup" Nov 03 02:30:20.408401 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.408399 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c076195c-71f0-49cc-8aef-b690cab08678" containerName="cleanup" Nov 03 02:30:20.408605 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.408408 5463 cpu_manager.go:414] "containerMap: removing container" podUID="e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f" containerName="cleanup" Nov 03 02:30:20.408605 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.408413 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f" containerName="cleanup" Nov 03 02:30:20.408605 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.408449 5463 cpu_manager.go:414] "containerMap: removing container" podUID="e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f" containerName="cleanup" Nov 03 02:30:20.408605 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.408454 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f" containerName="cleanup" Nov 03 02:30:20.408605 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.408464 5463 cpu_manager.go:414] "containerMap: removing container" podUID="0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98" containerName="cleanup" Nov 03 02:30:20.408605 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.408469 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ecbfe03-1eaf-4c5b-9f8b-62d5c88e3c98" containerName="cleanup" Nov 03 02:30:20.408605 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.408556 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c076195c-71f0-49cc-8aef-b690cab08678" containerName="cleanup" Nov 03 02:30:20.408605 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.408565 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c076195c-71f0-49cc-8aef-b690cab08678" containerName="cleanup" Nov 03 02:30:20.408605 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.408572 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f" containerName="cleanup" Nov 03 02:30:20.408605 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.408579 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c076195c-71f0-49cc-8aef-b690cab08678" containerName="cleanup" Nov 03 02:30:20.408605 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.408585 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f" containerName="cleanup" Nov 03 02:30:20.412177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.412127 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.467512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.465495 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c076195c-71f0-49cc-8aef-b690cab08678" path="/var/lib/kubelet/pods/c076195c-71f0-49cc-8aef-b690cab08678/volumes" Nov 03 02:30:20.467512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.465895 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d94af3-17e1-4b9e-b27e-1810694acc49-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.467512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.465995 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.467512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.466023 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.467512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.466057 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.467512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.466108 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.467512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.466170 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.467512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.466230 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.467512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.466264 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5j2m\" (UniqueName: \"kubernetes.io/projected/12d94af3-17e1-4b9e-b27e-1810694acc49-kube-api-access-r5j2m\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.467512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.466367 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.476732 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.471615 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d"] Nov 03 02:30:20.567370 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.567333 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.567370 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.567372 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5j2m\" (UniqueName: \"kubernetes.io/projected/12d94af3-17e1-4b9e-b27e-1810694acc49-kube-api-access-r5j2m\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.567601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.567393 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.567601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.567528 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d94af3-17e1-4b9e-b27e-1810694acc49-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.567601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.567577 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.567801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.567606 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.567801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.567640 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.567801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.567680 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.567801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.567740 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.568111 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.568084 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.568169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.568103 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.568274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.568248 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.568488 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.568407 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.568488 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.568407 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d94af3-17e1-4b9e-b27e-1810694acc49-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.570394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.570370 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.570501 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.570479 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.624563 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.624526 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.624691 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.624576 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5j2m\" (UniqueName: \"kubernetes.io/projected/12d94af3-17e1-4b9e-b27e-1810694acc49-kube-api-access-r5j2m\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.667482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.667406 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25" Nov 03 02:30:20.667482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.667431 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25" event={"ID":"e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f","Type":"ContainerDied","Data":"90c269ad55fd1e2c03ee7802a8e3ef6eec10a338de7668e1da945eb964830143"} Nov 03 02:30:20.667702 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.667503 5463 scope.go:122] "RemoveContainer" containerID="a80b17dd057a5edde0e3b23aae1f04ae6624bd3a1232114c3019025d7cb74dd3" Nov 03 02:30:20.726756 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.726719 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:20.805676 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.805640 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl"] Nov 03 02:30:20.837682 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.837627 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw" Nov 03 02:30:20.844159 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.844128 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25"] Nov 03 02:30:20.848496 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.848469 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-8857cb-29894550-wmk25"] Nov 03 02:30:20.881096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.880805 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-maas-api-xks-ca\") pod \"a8ba3dae-180f-4d19-a5d3-fb4dbf253295\" (UID: \"a8ba3dae-180f-4d19-a5d3-fb4dbf253295\") " Nov 03 02:30:20.881096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.880874 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-maas-api-openshift-ca\") pod \"a8ba3dae-180f-4d19-a5d3-fb4dbf253295\" (UID: \"a8ba3dae-180f-4d19-a5d3-fb4dbf253295\") " Nov 03 02:30:20.881096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.880915 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-kube-api-access-2rgdd\" (UniqueName: \"kubernetes.io/projected/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-kube-api-access-2rgdd\") pod \"a8ba3dae-180f-4d19-a5d3-fb4dbf253295\" (UID: \"a8ba3dae-180f-4d19-a5d3-fb4dbf253295\") " Nov 03 02:30:20.882407 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.882374 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-maas-api-openshift-ca" pod "a8ba3dae-180f-4d19-a5d3-fb4dbf253295" (UID: "a8ba3dae-180f-4d19-a5d3-fb4dbf253295"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:20.885518 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.885065 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-maas-api-xks-ca" pod "a8ba3dae-180f-4d19-a5d3-fb4dbf253295" (UID: "a8ba3dae-180f-4d19-a5d3-fb4dbf253295"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:30:20.886687 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.886660 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-kube-api-access-2rgdd" pod "a8ba3dae-180f-4d19-a5d3-fb4dbf253295" (UID: "a8ba3dae-180f-4d19-a5d3-fb4dbf253295"). InnerVolumeSpecName "kube-api-access-2rgdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:20.983271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.983183 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:20.983454 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.983440 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:20.983519 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.983511 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rgdd\" (UniqueName: \"kubernetes.io/projected/a8ba3dae-180f-4d19-a5d3-fb4dbf253295-kube-api-access-2rgdd\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:20.997423 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:20.997392 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl" Nov 03 02:30:21.084052 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.084024 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0c965f8-4f67-420c-9322-0afa9e745187-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c965f8-4f67-420c-9322-0afa9e745187-maas-api-service-ca\") pod \"c0c965f8-4f67-420c-9322-0afa9e745187\" (UID: \"c0c965f8-4f67-420c-9322-0afa9e745187\") " Nov 03 02:30:21.084242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.084220 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c965f8-4f67-420c-9322-0afa9e745187-kube-api-access-jvrtr\" (UniqueName: \"kubernetes.io/projected/c0c965f8-4f67-420c-9322-0afa9e745187-kube-api-access-jvrtr\") pod \"c0c965f8-4f67-420c-9322-0afa9e745187\" (UID: \"c0c965f8-4f67-420c-9322-0afa9e745187\") " Nov 03 02:30:21.084477 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.084450 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c965f8-4f67-420c-9322-0afa9e745187-maas-api-service-ca" pod "c0c965f8-4f67-420c-9322-0afa9e745187" (UID: "c0c965f8-4f67-420c-9322-0afa9e745187"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:21.084719 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.084702 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c965f8-4f67-420c-9322-0afa9e745187-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:21.086310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.086286 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c965f8-4f67-420c-9322-0afa9e745187-kube-api-access-jvrtr" pod "c0c965f8-4f67-420c-9322-0afa9e745187" (UID: "c0c965f8-4f67-420c-9322-0afa9e745187"). InnerVolumeSpecName "kube-api-access-jvrtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:21.179132 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:30:21.179099 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d94af3_17e1_4b9e_b27e_1810694acc49.slice/crio-f94d4e33b8f1cc14437d3cdb429cff610477e0f0b1fb9486aa9f361a2218ed5e WatchSource:0}: Error finding container f94d4e33b8f1cc14437d3cdb429cff610477e0f0b1fb9486aa9f361a2218ed5e: Status 404 returned error can't find the container with id f94d4e33b8f1cc14437d3cdb429cff610477e0f0b1fb9486aa9f361a2218ed5e Nov 03 02:30:21.184179 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.182750 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:30:21.184179 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.182826 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:30:21.184179 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.182876 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:30:21.184179 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.183491 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d"] Nov 03 02:30:21.187145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.187103 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvrtr\" (UniqueName: \"kubernetes.io/projected/c0c965f8-4f67-420c-9322-0afa9e745187-kube-api-access-jvrtr\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:21.342689 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.342657 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:21.678837 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.678804 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw" Nov 03 02:30:21.678837 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.678821 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw" event={"ID":"a8ba3dae-180f-4d19-a5d3-fb4dbf253295","Type":"ContainerDied","Data":"ecff6a397defd376919ea36c8c5756ada76c722c6e8c3bdce9867f222e59e28e"} Nov 03 02:30:21.679105 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.678875 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecff6a397defd376919ea36c8c5756ada76c722c6e8c3bdce9867f222e59e28e" Nov 03 02:30:21.680760 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.680726 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" event={"ID":"12d94af3-17e1-4b9e-b27e-1810694acc49","Type":"ContainerStarted","Data":"43c6716a3d00202658b6cabe79614d0f34f8aea848422fabf0bd41559d3a04af"} Nov 03 02:30:21.680923 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.680790 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" event={"ID":"12d94af3-17e1-4b9e-b27e-1810694acc49","Type":"ContainerStarted","Data":"f94d4e33b8f1cc14437d3cdb429cff610477e0f0b1fb9486aa9f361a2218ed5e"} Nov 03 02:30:21.682509 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.682227 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl" event={"ID":"c0c965f8-4f67-420c-9322-0afa9e745187","Type":"ContainerDied","Data":"ad082d0e94c99d2fc0c5367e1bcc602cf7eb197f6ffaa3c0cc35adcd634ba682"} Nov 03 02:30:21.682509 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.682290 5463 scope.go:122] "RemoveContainer" containerID="ccb5fe9c862cf5d9f58dd6be49f1289f5cdebb62b0331fcd28e437a2f4d31691" Nov 03 02:30:21.682509 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.682317 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl" Nov 03 02:30:21.727002 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.726981 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:21.828282 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.828185 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs"] Nov 03 02:30:21.828701 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.828463 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" podUID="a7d39617-aba6-4951-b3ce-14f544abbf7a" containerName="istio-proxy" containerID="cri-o://9aaf1ece518e1c2b57ec4658d0c7bd01954314ba84c4c9050d4c3b1e72d2cf03" gracePeriod=30 Nov 03 02:30:21.838692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.838646 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" podStartSLOduration=1.838632192 podStartE2EDuration="1.838632192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:21.835805197 +0000 UTC m=+2303.519914352" watchObservedRunningTime="2026-11-03 02:30:21.838632192 +0000 UTC m=+2303.522741427" Nov 03 02:30:21.862060 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.862027 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:21.966449 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.966351 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl"] Nov 03 02:30:21.971095 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:21.971066 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a0e72e-29894550-kqttl"] Nov 03 02:30:22.444338 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:22.444309 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0c965f8-4f67-420c-9322-0afa9e745187" path="/var/lib/kubelet/pods/c0c965f8-4f67-420c-9322-0afa9e745187/volumes" Nov 03 02:30:22.444755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:22.444742 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f" path="/var/lib/kubelet/pods/e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f/volumes" Nov 03 02:30:22.728633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:22.728562 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Nov 03 02:30:22.728633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:22.728610 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" podUID="12d94af3-17e1-4b9e-b27e-1810694acc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Nov 03 02:30:23.727908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:23.727864 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Nov 03 02:30:23.728380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:23.727926 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" podUID="12d94af3-17e1-4b9e-b27e-1810694acc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Nov 03 02:30:23.828641 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:23.828606 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:24.336373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:24.336343 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:24.732596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:24.732553 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Nov 03 02:30:24.733203 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:24.733148 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" podUID="12d94af3-17e1-4b9e-b27e-1810694acc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Nov 03 02:30:24.882225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:24.881175 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:25.412530 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.412499 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v"] Nov 03 02:30:25.413116 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.413103 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a8ba3dae-180f-4d19-a5d3-fb4dbf253295" containerName="revoke-keys" Nov 03 02:30:25.413170 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.413118 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ba3dae-180f-4d19-a5d3-fb4dbf253295" containerName="revoke-keys" Nov 03 02:30:25.413170 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.413141 5463 cpu_manager.go:414] "containerMap: removing container" podUID="c076195c-71f0-49cc-8aef-b690cab08678" containerName="cleanup" Nov 03 02:30:25.413170 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.413146 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c076195c-71f0-49cc-8aef-b690cab08678" containerName="cleanup" Nov 03 02:30:25.413170 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.413162 5463 cpu_manager.go:414] "containerMap: removing container" podUID="c0c965f8-4f67-420c-9322-0afa9e745187" containerName="cleanup" Nov 03 02:30:25.413170 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.413167 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c965f8-4f67-420c-9322-0afa9e745187" containerName="cleanup" Nov 03 02:30:25.413380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.413200 5463 cpu_manager.go:414] "containerMap: removing container" podUID="c0c965f8-4f67-420c-9322-0afa9e745187" containerName="cleanup" Nov 03 02:30:25.413380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.413205 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c965f8-4f67-420c-9322-0afa9e745187" containerName="cleanup" Nov 03 02:30:25.413380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.413218 5463 cpu_manager.go:414] "containerMap: removing container" podUID="c0c965f8-4f67-420c-9322-0afa9e745187" containerName="cleanup" Nov 03 02:30:25.413380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.413222 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c965f8-4f67-420c-9322-0afa9e745187" containerName="cleanup" Nov 03 02:30:25.413380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.413232 5463 cpu_manager.go:414] "containerMap: removing container" podUID="e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f" containerName="cleanup" Nov 03 02:30:25.413380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.413236 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f" containerName="cleanup" Nov 03 02:30:25.413380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.413322 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c965f8-4f67-420c-9322-0afa9e745187" containerName="cleanup" Nov 03 02:30:25.413380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.413329 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e672d1ee-fcfe-4f14-9d41-9ec7bbd2d82f" containerName="cleanup" Nov 03 02:30:25.413380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.413338 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c965f8-4f67-420c-9322-0afa9e745187" containerName="cleanup" Nov 03 02:30:25.413380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.413347 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ba3dae-180f-4d19-a5d3-fb4dbf253295" containerName="revoke-keys" Nov 03 02:30:25.416768 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.416746 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v" Nov 03 02:30:25.485279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.474578 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v"] Nov 03 02:30:25.516076 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.516044 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fb1940\"" Nov 03 02:30:25.516076 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.516044 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fb1940-dockercfg-5htss\"" Nov 03 02:30:25.586075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.586041 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x42nq\" (UniqueName: \"kubernetes.io/projected/14e3d223-26f5-43d9-9f48-1818d0bfe12e-kube-api-access-x42nq\") pod \"payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v\" (UID: \"14e3d223-26f5-43d9-9f48-1818d0bfe12e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v" Nov 03 02:30:25.586288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.586252 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e3d223-26f5-43d9-9f48-1818d0bfe12e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v\" (UID: \"14e3d223-26f5-43d9-9f48-1818d0bfe12e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v" Nov 03 02:30:25.626912 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.626869 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr"] Nov 03 02:30:25.687914 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.687826 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e3d223-26f5-43d9-9f48-1818d0bfe12e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v\" (UID: \"14e3d223-26f5-43d9-9f48-1818d0bfe12e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v" Nov 03 02:30:25.688074 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.687960 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42nq\" (UniqueName: \"kubernetes.io/projected/14e3d223-26f5-43d9-9f48-1818d0bfe12e-kube-api-access-x42nq\") pod \"payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v\" (UID: \"14e3d223-26f5-43d9-9f48-1818d0bfe12e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v" Nov 03 02:30:25.688816 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.688789 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e3d223-26f5-43d9-9f48-1818d0bfe12e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v\" (UID: \"14e3d223-26f5-43d9-9f48-1818d0bfe12e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v" Nov 03 02:30:25.728112 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.728081 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Nov 03 02:30:25.728309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.728149 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" podUID="12d94af3-17e1-4b9e-b27e-1810694acc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Nov 03 02:30:25.751214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.747736 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x42nq\" (UniqueName: \"kubernetes.io/projected/14e3d223-26f5-43d9-9f48-1818d0bfe12e-kube-api-access-x42nq\") pod \"payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v\" (UID: \"14e3d223-26f5-43d9-9f48-1818d0bfe12e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v" Nov 03 02:30:25.888504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:25.888471 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:26.031419 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.031328 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v" Nov 03 02:30:26.047831 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.047799 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:26.207234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.203067 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52"] Nov 03 02:30:26.207234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.204387 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c965f8-4f67-420c-9322-0afa9e745187" containerName="cleanup" Nov 03 02:30:26.209634 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.209612 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52" Nov 03 02:30:26.289316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.287431 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52"] Nov 03 02:30:26.299989 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.299776 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zznjv\" (UniqueName: \"kubernetes.io/projected/d9153ab8-8061-4275-b9ad-8aea6e949ab7-kube-api-access-zznjv\") pod \"payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52\" (UID: \"d9153ab8-8061-4275-b9ad-8aea6e949ab7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52" Nov 03 02:30:26.299989 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.299945 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9153ab8-8061-4275-b9ad-8aea6e949ab7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52\" (UID: \"d9153ab8-8061-4275-b9ad-8aea6e949ab7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52" Nov 03 02:30:26.401383 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.401352 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zznjv\" (UniqueName: \"kubernetes.io/projected/d9153ab8-8061-4275-b9ad-8aea6e949ab7-kube-api-access-zznjv\") pod \"payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52\" (UID: \"d9153ab8-8061-4275-b9ad-8aea6e949ab7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52" Nov 03 02:30:26.401591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.401434 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9153ab8-8061-4275-b9ad-8aea6e949ab7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52\" (UID: \"d9153ab8-8061-4275-b9ad-8aea6e949ab7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52" Nov 03 02:30:26.402286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.402261 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9153ab8-8061-4275-b9ad-8aea6e949ab7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52\" (UID: \"d9153ab8-8061-4275-b9ad-8aea6e949ab7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52" Nov 03 02:30:26.493505 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.493463 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zznjv\" (UniqueName: \"kubernetes.io/projected/d9153ab8-8061-4275-b9ad-8aea6e949ab7-kube-api-access-zznjv\") pod \"payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52\" (UID: \"d9153ab8-8061-4275-b9ad-8aea6e949ab7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52" Nov 03 02:30:26.496422 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.496389 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v"] Nov 03 02:30:26.525395 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.525372 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52" Nov 03 02:30:26.727504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.727472 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Nov 03 02:30:26.727657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.727522 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" podUID="12d94af3-17e1-4b9e-b27e-1810694acc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Nov 03 02:30:26.747317 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.747281 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v" event={"ID":"14e3d223-26f5-43d9-9f48-1818d0bfe12e","Type":"ContainerStarted","Data":"b08a9f479ac5100bb210533950056ac7412a36eb89cfdf4ffd0d695072d34b71"} Nov 03 02:30:26.747563 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.747543 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v" event={"ID":"14e3d223-26f5-43d9-9f48-1818d0bfe12e","Type":"ContainerStarted","Data":"f130566df0a5ae6eaa3bb8f5e397d42a69348664e4ae4d0f560e4aa84950e9ed"} Nov 03 02:30:26.747729 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.747712 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v" Nov 03 02:30:26.938725 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:30:26.938685 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9153ab8_8061_4275_b9ad_8aea6e949ab7.slice/crio-5974272fbd7db50d6f4bc24dd509b37c14ca349e338260fb598baf3484b678d2 WatchSource:0}: Error finding container 5974272fbd7db50d6f4bc24dd509b37c14ca349e338260fb598baf3484b678d2: Status 404 returned error can't find the container with id 5974272fbd7db50d6f4bc24dd509b37c14ca349e338260fb598baf3484b678d2 Nov 03 02:30:26.941526 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.941480 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v" podStartSLOduration=1.9414658889999998 podStartE2EDuration="1.941465889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:26.873550949 +0000 UTC m=+2308.557660105" watchObservedRunningTime="2026-11-03 02:30:26.941465889 +0000 UTC m=+2308.625575090" Nov 03 02:30:26.942596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:26.942570 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52"] Nov 03 02:30:27.338360 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.338324 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:27.428101 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.428080 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:30:27.530158 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.530119 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-workload-certs\") pod \"a7d39617-aba6-4951-b3ce-14f544abbf7a\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " Nov 03 02:30:27.530349 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.530241 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-token\" (UniqueName: \"kubernetes.io/projected/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-token\") pod \"a7d39617-aba6-4951-b3ce-14f544abbf7a\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " Nov 03 02:30:27.530349 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.530280 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-credential-socket\") pod \"a7d39617-aba6-4951-b3ce-14f544abbf7a\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " Nov 03 02:30:27.530349 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.530313 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-data\") pod \"a7d39617-aba6-4951-b3ce-14f544abbf7a\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " Nov 03 02:30:27.530349 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.530338 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-podinfo\") pod \"a7d39617-aba6-4951-b3ce-14f544abbf7a\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " Nov 03 02:30:27.530487 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.530431 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d39617-aba6-4951-b3ce-14f544abbf7a-kube-api-access-6smsm\" (UniqueName: \"kubernetes.io/projected/a7d39617-aba6-4951-b3ce-14f544abbf7a-kube-api-access-6smsm\") pod \"a7d39617-aba6-4951-b3ce-14f544abbf7a\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " Nov 03 02:30:27.530520 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.530502 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7d39617-aba6-4951-b3ce-14f544abbf7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d39617-aba6-4951-b3ce-14f544abbf7a-istiod-ca-cert\") pod \"a7d39617-aba6-4951-b3ce-14f544abbf7a\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " Nov 03 02:30:27.530556 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.530532 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-workload-socket\") pod \"a7d39617-aba6-4951-b3ce-14f544abbf7a\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " Nov 03 02:30:27.530593 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.530561 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-envoy\") pod \"a7d39617-aba6-4951-b3ce-14f544abbf7a\" (UID: \"a7d39617-aba6-4951-b3ce-14f544abbf7a\") " Nov 03 02:30:27.530650 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.530616 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-credential-socket" pod "a7d39617-aba6-4951-b3ce-14f544abbf7a" (UID: "a7d39617-aba6-4951-b3ce-14f544abbf7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:30:27.530787 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.530762 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-workload-certs" pod "a7d39617-aba6-4951-b3ce-14f544abbf7a" (UID: "a7d39617-aba6-4951-b3ce-14f544abbf7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:30:27.531069 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.530966 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-data" pod "a7d39617-aba6-4951-b3ce-14f544abbf7a" (UID: "a7d39617-aba6-4951-b3ce-14f544abbf7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:30:27.531069 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.531056 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d39617-aba6-4951-b3ce-14f544abbf7a-istiod-ca-cert" pod "a7d39617-aba6-4951-b3ce-14f544abbf7a" (UID: "a7d39617-aba6-4951-b3ce-14f544abbf7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:27.531212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.531111 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-workload-socket" pod "a7d39617-aba6-4951-b3ce-14f544abbf7a" (UID: "a7d39617-aba6-4951-b3ce-14f544abbf7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:30:27.531609 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.531526 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:27.531705 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.531615 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:27.531705 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.531634 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:27.531705 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.531647 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d39617-aba6-4951-b3ce-14f544abbf7a-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:27.531705 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.531659 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:27.533129 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.533105 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-token" pod "a7d39617-aba6-4951-b3ce-14f544abbf7a" (UID: "a7d39617-aba6-4951-b3ce-14f544abbf7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:27.533423 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.533402 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-podinfo" pod "a7d39617-aba6-4951-b3ce-14f544abbf7a" (UID: "a7d39617-aba6-4951-b3ce-14f544abbf7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:30:27.533538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.533520 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d39617-aba6-4951-b3ce-14f544abbf7a-kube-api-access-6smsm" pod "a7d39617-aba6-4951-b3ce-14f544abbf7a" (UID: "a7d39617-aba6-4951-b3ce-14f544abbf7a"). InnerVolumeSpecName "kube-api-access-6smsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:27.533782 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.533759 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-envoy" pod "a7d39617-aba6-4951-b3ce-14f544abbf7a" (UID: "a7d39617-aba6-4951-b3ce-14f544abbf7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:30:27.634215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.633248 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:27.634215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.633282 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:27.634215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.633295 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6smsm\" (UniqueName: \"kubernetes.io/projected/a7d39617-aba6-4951-b3ce-14f544abbf7a-kube-api-access-6smsm\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:27.634215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.633309 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d39617-aba6-4951-b3ce-14f544abbf7a-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:27.634215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.634030 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:27.727950 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.727866 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Nov 03 02:30:27.727950 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.727931 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" podUID="12d94af3-17e1-4b9e-b27e-1810694acc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Nov 03 02:30:27.766644 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.765508 5463 generic.go:356] "Generic (PLEG): container finished" podID="a7d39617-aba6-4951-b3ce-14f544abbf7a" containerID="9aaf1ece518e1c2b57ec4658d0c7bd01954314ba84c4c9050d4c3b1e72d2cf03" exitCode=0 Nov 03 02:30:27.766644 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.766084 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" Nov 03 02:30:27.796814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.772682 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" event={"ID":"a7d39617-aba6-4951-b3ce-14f544abbf7a","Type":"ContainerDied","Data":"9aaf1ece518e1c2b57ec4658d0c7bd01954314ba84c4c9050d4c3b1e72d2cf03"} Nov 03 02:30:27.796814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.772745 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs" event={"ID":"a7d39617-aba6-4951-b3ce-14f544abbf7a","Type":"ContainerDied","Data":"8bd817560f5f8808e3a76853c06e6d62521ec5c2cfeb892de80030324e4b6deb"} Nov 03 02:30:27.796814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.772775 5463 scope.go:122] "RemoveContainer" containerID="9aaf1ece518e1c2b57ec4658d0c7bd01954314ba84c4c9050d4c3b1e72d2cf03" Nov 03 02:30:27.796814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.793240 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52" event={"ID":"d9153ab8-8061-4275-b9ad-8aea6e949ab7","Type":"ContainerStarted","Data":"36be79d2dd5147f499b052916c1061cb364d403a09f1bfcd46f0bb53c3283b02"} Nov 03 02:30:27.796814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.793392 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52" Nov 03 02:30:27.796814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.793422 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52" event={"ID":"d9153ab8-8061-4275-b9ad-8aea6e949ab7","Type":"ContainerStarted","Data":"5974272fbd7db50d6f4bc24dd509b37c14ca349e338260fb598baf3484b678d2"} Nov 03 02:30:27.813716 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.812792 5463 scope.go:122] "RemoveContainer" containerID="42e9941f147f6c7e61d605cdffa98cc6a0b7707baa76c6853888e3011bd2fa7e" Nov 03 02:30:27.827375 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.827350 5463 scope.go:122] "RemoveContainer" containerID="9aaf1ece518e1c2b57ec4658d0c7bd01954314ba84c4c9050d4c3b1e72d2cf03" Nov 03 02:30:27.829564 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:30:27.829533 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aaf1ece518e1c2b57ec4658d0c7bd01954314ba84c4c9050d4c3b1e72d2cf03\": container with ID starting with 9aaf1ece518e1c2b57ec4658d0c7bd01954314ba84c4c9050d4c3b1e72d2cf03 not found: ID does not exist" containerID="9aaf1ece518e1c2b57ec4658d0c7bd01954314ba84c4c9050d4c3b1e72d2cf03" Nov 03 02:30:27.829693 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.829581 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aaf1ece518e1c2b57ec4658d0c7bd01954314ba84c4c9050d4c3b1e72d2cf03"} err="failed to get container status \"9aaf1ece518e1c2b57ec4658d0c7bd01954314ba84c4c9050d4c3b1e72d2cf03\": rpc error: code = NotFound desc = could not find container \"9aaf1ece518e1c2b57ec4658d0c7bd01954314ba84c4c9050d4c3b1e72d2cf03\": container with ID starting with 9aaf1ece518e1c2b57ec4658d0c7bd01954314ba84c4c9050d4c3b1e72d2cf03 not found: ID does not exist" Nov 03 02:30:27.829693 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.829607 5463 scope.go:122] "RemoveContainer" containerID="42e9941f147f6c7e61d605cdffa98cc6a0b7707baa76c6853888e3011bd2fa7e" Nov 03 02:30:27.829988 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:30:27.829919 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e9941f147f6c7e61d605cdffa98cc6a0b7707baa76c6853888e3011bd2fa7e\": container with ID starting with 42e9941f147f6c7e61d605cdffa98cc6a0b7707baa76c6853888e3011bd2fa7e not found: ID does not exist" containerID="42e9941f147f6c7e61d605cdffa98cc6a0b7707baa76c6853888e3011bd2fa7e" Nov 03 02:30:27.829988 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.829959 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e9941f147f6c7e61d605cdffa98cc6a0b7707baa76c6853888e3011bd2fa7e"} err="failed to get container status \"42e9941f147f6c7e61d605cdffa98cc6a0b7707baa76c6853888e3011bd2fa7e\": rpc error: code = NotFound desc = could not find container \"42e9941f147f6c7e61d605cdffa98cc6a0b7707baa76c6853888e3011bd2fa7e\": container with ID starting with 42e9941f147f6c7e61d605cdffa98cc6a0b7707baa76c6853888e3011bd2fa7e not found: ID does not exist" Nov 03 02:30:27.943741 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.943708 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs"] Nov 03 02:30:27.951214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:27.947587 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-56299c-openshift-default-75d88fccf9-zc6hs"] Nov 03 02:30:28.064219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:28.063051 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52" podStartSLOduration=2.063037567 podStartE2EDuration="2.063037567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:28.060693122 +0000 UTC m=+2309.744802291" watchObservedRunningTime="2026-11-03 02:30:28.063037567 +0000 UTC m=+2309.747146721" Nov 03 02:30:28.445581 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:28.445548 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d39617-aba6-4951-b3ce-14f544abbf7a" path="/var/lib/kubelet/pods/a7d39617-aba6-4951-b3ce-14f544abbf7a/volumes" Nov 03 02:30:28.727324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:28.727250 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Nov 03 02:30:28.727324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:28.727312 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" podUID="12d94af3-17e1-4b9e-b27e-1810694acc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Nov 03 02:30:29.731564 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:29.731533 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:29.731980 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:29.731775 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:29.732472 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:29.732453 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:30:30.627996 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:30.627924 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr" podUID="284b69a9-1ced-4c2f-9698-7682e921ca69" containerName="payload-pre-processing" containerID="cri-o://53e11a75a19c1ee73028ed559c380ea2165876775d1d42736df5b512ba240a6a" gracePeriod=25 Nov 03 02:30:30.728517 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:30.728469 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:30.833279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:30.833043 5463 generic.go:356] "Generic (PLEG): container finished" podID="284b69a9-1ced-4c2f-9698-7682e921ca69" containerID="53e11a75a19c1ee73028ed559c380ea2165876775d1d42736df5b512ba240a6a" exitCode=0 Nov 03 02:30:30.833279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:30.833110 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr" event={"ID":"284b69a9-1ced-4c2f-9698-7682e921ca69","Type":"ContainerDied","Data":"53e11a75a19c1ee73028ed559c380ea2165876775d1d42736df5b512ba240a6a"} Nov 03 02:30:30.899340 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:30.899312 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr" Nov 03 02:30:30.975530 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:30.975492 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/284b69a9-1ced-4c2f-9698-7682e921ca69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284b69a9-1ced-4c2f-9698-7682e921ca69-plugins-config-volume\") pod \"284b69a9-1ced-4c2f-9698-7682e921ca69\" (UID: \"284b69a9-1ced-4c2f-9698-7682e921ca69\") " Nov 03 02:30:30.975704 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:30.975543 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284b69a9-1ced-4c2f-9698-7682e921ca69-kube-api-access-w8zjg\" (UniqueName: \"kubernetes.io/projected/284b69a9-1ced-4c2f-9698-7682e921ca69-kube-api-access-w8zjg\") pod \"284b69a9-1ced-4c2f-9698-7682e921ca69\" (UID: \"284b69a9-1ced-4c2f-9698-7682e921ca69\") " Nov 03 02:30:30.975926 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:30.975896 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/284b69a9-1ced-4c2f-9698-7682e921ca69-plugins-config-volume" pod "284b69a9-1ced-4c2f-9698-7682e921ca69" (UID: "284b69a9-1ced-4c2f-9698-7682e921ca69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:30.976139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:30.976117 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284b69a9-1ced-4c2f-9698-7682e921ca69-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:30.978133 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:30.978109 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284b69a9-1ced-4c2f-9698-7682e921ca69-kube-api-access-w8zjg" pod "284b69a9-1ced-4c2f-9698-7682e921ca69" (UID: "284b69a9-1ced-4c2f-9698-7682e921ca69"). InnerVolumeSpecName "kube-api-access-w8zjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:31.077763 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:31.077729 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8zjg\" (UniqueName: \"kubernetes.io/projected/284b69a9-1ced-4c2f-9698-7682e921ca69-kube-api-access-w8zjg\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:31.843077 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:31.843050 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr" Nov 03 02:30:31.843546 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:31.843087 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr" event={"ID":"284b69a9-1ced-4c2f-9698-7682e921ca69","Type":"ContainerDied","Data":"e305d04a2c5eb4885ba5e281a7a9915942338eb2df8bba8aba364a9cff5fb8b2"} Nov 03 02:30:31.843546 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:31.843146 5463 scope.go:122] "RemoveContainer" containerID="53e11a75a19c1ee73028ed559c380ea2165876775d1d42736df5b512ba240a6a" Nov 03 02:30:32.007564 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:32.007532 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr"] Nov 03 02:30:32.057786 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:32.057759 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f760d3-69c5b68875-5cbdr"] Nov 03 02:30:32.445610 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:32.445574 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="284b69a9-1ced-4c2f-9698-7682e921ca69" path="/var/lib/kubelet/pods/284b69a9-1ced-4c2f-9698-7682e921ca69/volumes" Nov 03 02:30:33.513771 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:33.513738 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:33.995926 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:33.995893 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:34.169381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:34.169344 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:34.313124 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:34.313045 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:37.796021 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:37.795993 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v" Nov 03 02:30:38.808014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:38.807985 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52" Nov 03 02:30:39.312445 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:39.312410 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:43.007998 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:30:43.007962 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:30:44.512944 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:44.512914 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:49.614018 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:49.613980 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:54.758246 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:54.758182 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:57.356667 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:57.355991 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": read tcp 10.133.0.2:37668->10.133.0.204:15021: read: connection reset by peer" start-of-body= Nov 03 02:30:57.356667 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:57.356071 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" podUID="da4fd375-fcd1-4665-8a2e-89a67bb9dc43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": read tcp 10.133.0.2:37668->10.133.0.204:15021: read: connection reset by peer" Nov 03 02:30:58.085556 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:58.085523 5463 generic.go:356] "Generic (PLEG): container finished" podID="da4fd375-fcd1-4665-8a2e-89a67bb9dc43" containerID="4b6e748def28d2bc8156b0730cc50284d393f153af9ed4bee58ea6d6050e1dbf" exitCode=0 Nov 03 02:30:58.085728 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:58.085561 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" event={"ID":"da4fd375-fcd1-4665-8a2e-89a67bb9dc43","Type":"ContainerDied","Data":"4b6e748def28d2bc8156b0730cc50284d393f153af9ed4bee58ea6d6050e1dbf"} Nov 03 02:30:58.085914 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:58.085897 5463 scope.go:122] "RemoveContainer" containerID="4b6e748def28d2bc8156b0730cc50284d393f153af9ed4bee58ea6d6050e1dbf" Nov 03 02:30:58.087376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:58.087344 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:30:58.087469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:58.087416 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:30:58.087469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:58.087448 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:30:59.096483 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:59.096440 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" event={"ID":"da4fd375-fcd1-4665-8a2e-89a67bb9dc43","Type":"ContainerStarted","Data":"9d4857cba5b69b46689d1c572dc201c474aaa59e3a5a1c33c9755bc5ceb5f7c1"} Nov 03 02:30:59.225800 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:59.225766 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t"] Nov 03 02:30:59.226047 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:59.226023 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" podUID="5d814d4b-c428-47d5-a594-578ba0b6bfd9" containerName="maas-api" containerID="cri-o://39f8272e37db55f612249805bbc7facfe3463dcae6fac1fb5d749400b278af0a" gracePeriod=30 Nov 03 02:30:59.343155 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:59.343124 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:59.581670 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:59.581643 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" Nov 03 02:30:59.698674 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:59.697859 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d814d4b-c428-47d5-a594-578ba0b6bfd9-kube-api-access-wrlql\" (UniqueName: \"kubernetes.io/projected/5d814d4b-c428-47d5-a594-578ba0b6bfd9-kube-api-access-wrlql\") pod \"5d814d4b-c428-47d5-a594-578ba0b6bfd9\" (UID: \"5d814d4b-c428-47d5-a594-578ba0b6bfd9\") " Nov 03 02:30:59.698674 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:59.697967 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d814d4b-c428-47d5-a594-578ba0b6bfd9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d814d4b-c428-47d5-a594-578ba0b6bfd9-maas-api-tls\") pod \"5d814d4b-c428-47d5-a594-578ba0b6bfd9\" (UID: \"5d814d4b-c428-47d5-a594-578ba0b6bfd9\") " Nov 03 02:30:59.698674 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:59.698041 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d814d4b-c428-47d5-a594-578ba0b6bfd9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d814d4b-c428-47d5-a594-578ba0b6bfd9-metrics-certs\") pod \"5d814d4b-c428-47d5-a594-578ba0b6bfd9\" (UID: \"5d814d4b-c428-47d5-a594-578ba0b6bfd9\") " Nov 03 02:30:59.698674 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:59.698118 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d814d4b-c428-47d5-a594-578ba0b6bfd9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d814d4b-c428-47d5-a594-578ba0b6bfd9-openshift-service-ca\") pod \"5d814d4b-c428-47d5-a594-578ba0b6bfd9\" (UID: \"5d814d4b-c428-47d5-a594-578ba0b6bfd9\") " Nov 03 02:30:59.699135 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:59.699102 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d814d4b-c428-47d5-a594-578ba0b6bfd9-openshift-service-ca" pod "5d814d4b-c428-47d5-a594-578ba0b6bfd9" (UID: "5d814d4b-c428-47d5-a594-578ba0b6bfd9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:59.701302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:59.701268 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d814d4b-c428-47d5-a594-578ba0b6bfd9-kube-api-access-wrlql" pod "5d814d4b-c428-47d5-a594-578ba0b6bfd9" (UID: "5d814d4b-c428-47d5-a594-578ba0b6bfd9"). InnerVolumeSpecName "kube-api-access-wrlql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:59.701891 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:59.701860 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d814d4b-c428-47d5-a594-578ba0b6bfd9-metrics-certs" pod "5d814d4b-c428-47d5-a594-578ba0b6bfd9" (UID: "5d814d4b-c428-47d5-a594-578ba0b6bfd9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:30:59.702105 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:59.702077 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d814d4b-c428-47d5-a594-578ba0b6bfd9-maas-api-tls" pod "5d814d4b-c428-47d5-a594-578ba0b6bfd9" (UID: "5d814d4b-c428-47d5-a594-578ba0b6bfd9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:30:59.799804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:59.799768 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrlql\" (UniqueName: \"kubernetes.io/projected/5d814d4b-c428-47d5-a594-578ba0b6bfd9-kube-api-access-wrlql\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:59.799804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:59.799796 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d814d4b-c428-47d5-a594-578ba0b6bfd9-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:59.799804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:59.799807 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d814d4b-c428-47d5-a594-578ba0b6bfd9-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:59.800114 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:59.799816 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d814d4b-c428-47d5-a594-578ba0b6bfd9-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:59.945469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:59.945436 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:30:59.945659 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:59.945611 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Nov 03 02:30:59.945764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:30:59.945659 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" podUID="da4fd375-fcd1-4665-8a2e-89a67bb9dc43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Nov 03 02:31:00.104668 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:00.104569 5463 generic.go:356] "Generic (PLEG): container finished" podID="5d814d4b-c428-47d5-a594-578ba0b6bfd9" containerID="39f8272e37db55f612249805bbc7facfe3463dcae6fac1fb5d749400b278af0a" exitCode=0 Nov 03 02:31:00.104668 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:00.104623 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" Nov 03 02:31:00.104668 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:00.104651 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" event={"ID":"5d814d4b-c428-47d5-a594-578ba0b6bfd9","Type":"ContainerDied","Data":"39f8272e37db55f612249805bbc7facfe3463dcae6fac1fb5d749400b278af0a"} Nov 03 02:31:00.105321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:00.104743 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t" event={"ID":"5d814d4b-c428-47d5-a594-578ba0b6bfd9","Type":"ContainerDied","Data":"ec6b811f78618a2ae825f3dc0c55fcfed3e3cd0362d8c0fa07cc9eff7b8df7c4"} Nov 03 02:31:00.105321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:00.104779 5463 scope.go:122] "RemoveContainer" containerID="39f8272e37db55f612249805bbc7facfe3463dcae6fac1fb5d749400b278af0a" Nov 03 02:31:00.120509 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:00.120491 5463 scope.go:122] "RemoveContainer" containerID="39f8272e37db55f612249805bbc7facfe3463dcae6fac1fb5d749400b278af0a" Nov 03 02:31:00.120793 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:31:00.120776 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f8272e37db55f612249805bbc7facfe3463dcae6fac1fb5d749400b278af0a\": container with ID starting with 39f8272e37db55f612249805bbc7facfe3463dcae6fac1fb5d749400b278af0a not found: ID does not exist" containerID="39f8272e37db55f612249805bbc7facfe3463dcae6fac1fb5d749400b278af0a" Nov 03 02:31:00.120843 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:00.120800 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f8272e37db55f612249805bbc7facfe3463dcae6fac1fb5d749400b278af0a"} err="failed to get container status \"39f8272e37db55f612249805bbc7facfe3463dcae6fac1fb5d749400b278af0a\": rpc error: code = NotFound desc = could not find container \"39f8272e37db55f612249805bbc7facfe3463dcae6fac1fb5d749400b278af0a\": container with ID starting with 39f8272e37db55f612249805bbc7facfe3463dcae6fac1fb5d749400b278af0a not found: ID does not exist" Nov 03 02:31:00.290026 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:00.289990 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t"] Nov 03 02:31:00.293905 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:00.293869 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f760d3-76f8484c7-vw26t"] Nov 03 02:31:00.444788 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:00.444759 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d814d4b-c428-47d5-a594-578ba0b6bfd9" path="/var/lib/kubelet/pods/5d814d4b-c428-47d5-a594-578ba0b6bfd9/volumes" Nov 03 02:31:00.946096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:00.946062 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Nov 03 02:31:00.946339 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:00.946141 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" podUID="da4fd375-fcd1-4665-8a2e-89a67bb9dc43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Nov 03 02:31:01.279856 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:01.279755 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:01.724373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:01.724340 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:01.897337 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:01.897304 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:01.945703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:01.945664 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Nov 03 02:31:01.945901 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:01.945727 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" podUID="da4fd375-fcd1-4665-8a2e-89a67bb9dc43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Nov 03 02:31:02.946023 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:02.945986 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Nov 03 02:31:02.946540 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:02.946050 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" podUID="da4fd375-fcd1-4665-8a2e-89a67bb9dc43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Nov 03 02:31:03.231341 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:03.231256 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:03.727118 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:03.727086 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd"] Nov 03 02:31:03.727339 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:03.727315 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" podUID="d029bf9f-23a3-47c7-8eca-e343baed7d99" containerName="istio-proxy" containerID="cri-o://99cfafc85940344d8c897518cbc3a4442ffc8f6ab01716f2e1f713ab98543008" gracePeriod=30 Nov 03 02:31:03.742061 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:03.742038 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:03.946579 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:03.946545 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Nov 03 02:31:03.946995 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:03.946597 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" podUID="da4fd375-fcd1-4665-8a2e-89a67bb9dc43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Nov 03 02:31:04.582483 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:04.582449 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:04.950801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:04.950771 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:31:04.951312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:04.951050 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:31:04.952086 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:04.952061 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:31:05.719360 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:05.719328 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:06.419902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:06.419842 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:06.666509 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:06.666470 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:31:06.666674 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:06.666532 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" podUID="d029bf9f-23a3-47c7-8eca-e343baed7d99" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:31:09.012310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.012248 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:31:09.146029 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.146002 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-envoy\") pod \"d029bf9f-23a3-47c7-8eca-e343baed7d99\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " Nov 03 02:31:09.146171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.146106 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-data\") pod \"d029bf9f-23a3-47c7-8eca-e343baed7d99\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " Nov 03 02:31:09.146171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.146133 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-workload-socket\") pod \"d029bf9f-23a3-47c7-8eca-e343baed7d99\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " Nov 03 02:31:09.146171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.146155 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-podinfo\") pod \"d029bf9f-23a3-47c7-8eca-e343baed7d99\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " Nov 03 02:31:09.146365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.146209 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d029bf9f-23a3-47c7-8eca-e343baed7d99-kube-api-access-q74l7\" (UniqueName: \"kubernetes.io/projected/d029bf9f-23a3-47c7-8eca-e343baed7d99-kube-api-access-q74l7\") pod \"d029bf9f-23a3-47c7-8eca-e343baed7d99\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " Nov 03 02:31:09.146365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.146246 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-credential-socket\") pod \"d029bf9f-23a3-47c7-8eca-e343baed7d99\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " Nov 03 02:31:09.146365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.146304 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d029bf9f-23a3-47c7-8eca-e343baed7d99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d029bf9f-23a3-47c7-8eca-e343baed7d99-istiod-ca-cert\") pod \"d029bf9f-23a3-47c7-8eca-e343baed7d99\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " Nov 03 02:31:09.146365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.146344 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-workload-certs\") pod \"d029bf9f-23a3-47c7-8eca-e343baed7d99\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " Nov 03 02:31:09.146560 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.146397 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-token\" (UniqueName: \"kubernetes.io/projected/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-token\") pod \"d029bf9f-23a3-47c7-8eca-e343baed7d99\" (UID: \"d029bf9f-23a3-47c7-8eca-e343baed7d99\") " Nov 03 02:31:09.146560 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.146538 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-workload-socket" pod "d029bf9f-23a3-47c7-8eca-e343baed7d99" (UID: "d029bf9f-23a3-47c7-8eca-e343baed7d99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:31:09.146665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.146636 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-data" pod "d029bf9f-23a3-47c7-8eca-e343baed7d99" (UID: "d029bf9f-23a3-47c7-8eca-e343baed7d99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:31:09.146720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.146677 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-workload-certs" pod "d029bf9f-23a3-47c7-8eca-e343baed7d99" (UID: "d029bf9f-23a3-47c7-8eca-e343baed7d99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:31:09.146847 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.146819 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d029bf9f-23a3-47c7-8eca-e343baed7d99-istiod-ca-cert" pod "d029bf9f-23a3-47c7-8eca-e343baed7d99" (UID: "d029bf9f-23a3-47c7-8eca-e343baed7d99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:31:09.147327 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.147297 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-credential-socket" pod "d029bf9f-23a3-47c7-8eca-e343baed7d99" (UID: "d029bf9f-23a3-47c7-8eca-e343baed7d99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:31:09.147452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.147333 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:09.147452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.147357 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:09.147452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.147372 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d029bf9f-23a3-47c7-8eca-e343baed7d99-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:09.147452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.147386 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:09.148800 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.148772 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-envoy" pod "d029bf9f-23a3-47c7-8eca-e343baed7d99" (UID: "d029bf9f-23a3-47c7-8eca-e343baed7d99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:31:09.149313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.149283 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d029bf9f-23a3-47c7-8eca-e343baed7d99-kube-api-access-q74l7" pod "d029bf9f-23a3-47c7-8eca-e343baed7d99" (UID: "d029bf9f-23a3-47c7-8eca-e343baed7d99"). InnerVolumeSpecName "kube-api-access-q74l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:09.149313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.149296 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-podinfo" pod "d029bf9f-23a3-47c7-8eca-e343baed7d99" (UID: "d029bf9f-23a3-47c7-8eca-e343baed7d99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:31:09.149463 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.149400 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-token" pod "d029bf9f-23a3-47c7-8eca-e343baed7d99" (UID: "d029bf9f-23a3-47c7-8eca-e343baed7d99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:09.178160 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.178131 5463 generic.go:356] "Generic (PLEG): container finished" podID="d029bf9f-23a3-47c7-8eca-e343baed7d99" containerID="99cfafc85940344d8c897518cbc3a4442ffc8f6ab01716f2e1f713ab98543008" exitCode=0 Nov 03 02:31:09.178365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.178203 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" Nov 03 02:31:09.178365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.178224 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" event={"ID":"d029bf9f-23a3-47c7-8eca-e343baed7d99","Type":"ContainerDied","Data":"99cfafc85940344d8c897518cbc3a4442ffc8f6ab01716f2e1f713ab98543008"} Nov 03 02:31:09.178365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.178295 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd" event={"ID":"d029bf9f-23a3-47c7-8eca-e343baed7d99","Type":"ContainerDied","Data":"bd7fb9d2ec83abf4fc9425b7d070b8560027ae0059344634f2510a59b868cefd"} Nov 03 02:31:09.178365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.178326 5463 scope.go:122] "RemoveContainer" containerID="99cfafc85940344d8c897518cbc3a4442ffc8f6ab01716f2e1f713ab98543008" Nov 03 02:31:09.188873 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.188858 5463 scope.go:122] "RemoveContainer" containerID="93e1f9bbae83197112663116fd74171229e18aef93fe1ea898fbd10f5b7296fb" Nov 03 02:31:09.200966 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.200951 5463 scope.go:122] "RemoveContainer" containerID="99cfafc85940344d8c897518cbc3a4442ffc8f6ab01716f2e1f713ab98543008" Nov 03 02:31:09.201253 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:31:09.201235 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99cfafc85940344d8c897518cbc3a4442ffc8f6ab01716f2e1f713ab98543008\": container with ID starting with 99cfafc85940344d8c897518cbc3a4442ffc8f6ab01716f2e1f713ab98543008 not found: ID does not exist" containerID="99cfafc85940344d8c897518cbc3a4442ffc8f6ab01716f2e1f713ab98543008" Nov 03 02:31:09.201305 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.201260 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99cfafc85940344d8c897518cbc3a4442ffc8f6ab01716f2e1f713ab98543008"} err="failed to get container status \"99cfafc85940344d8c897518cbc3a4442ffc8f6ab01716f2e1f713ab98543008\": rpc error: code = NotFound desc = could not find container \"99cfafc85940344d8c897518cbc3a4442ffc8f6ab01716f2e1f713ab98543008\": container with ID starting with 99cfafc85940344d8c897518cbc3a4442ffc8f6ab01716f2e1f713ab98543008 not found: ID does not exist" Nov 03 02:31:09.201305 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.201279 5463 scope.go:122] "RemoveContainer" containerID="93e1f9bbae83197112663116fd74171229e18aef93fe1ea898fbd10f5b7296fb" Nov 03 02:31:09.201507 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:31:09.201493 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93e1f9bbae83197112663116fd74171229e18aef93fe1ea898fbd10f5b7296fb\": container with ID starting with 93e1f9bbae83197112663116fd74171229e18aef93fe1ea898fbd10f5b7296fb not found: ID does not exist" containerID="93e1f9bbae83197112663116fd74171229e18aef93fe1ea898fbd10f5b7296fb" Nov 03 02:31:09.201551 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.201515 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93e1f9bbae83197112663116fd74171229e18aef93fe1ea898fbd10f5b7296fb"} err="failed to get container status \"93e1f9bbae83197112663116fd74171229e18aef93fe1ea898fbd10f5b7296fb\": rpc error: code = NotFound desc = could not find container \"93e1f9bbae83197112663116fd74171229e18aef93fe1ea898fbd10f5b7296fb\": container with ID starting with 93e1f9bbae83197112663116fd74171229e18aef93fe1ea898fbd10f5b7296fb not found: ID does not exist" Nov 03 02:31:09.248784 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.248737 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:09.248784 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.248759 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:09.248784 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.248769 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q74l7\" (UniqueName: \"kubernetes.io/projected/d029bf9f-23a3-47c7-8eca-e343baed7d99-kube-api-access-q74l7\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:09.248784 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.248779 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d029bf9f-23a3-47c7-8eca-e343baed7d99-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:09.248973 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.248793 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d029bf9f-23a3-47c7-8eca-e343baed7d99-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:09.344549 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.344519 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd"] Nov 03 02:31:09.347382 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.347353 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-f760d3-openshift-default-84cd455b89-pqtzd"] Nov 03 02:31:09.925695 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:09.925661 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg"] Nov 03 02:31:10.199894 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:10.199808 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:10.357619 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:10.357586 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:10.443446 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:10.443412 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d029bf9f-23a3-47c7-8eca-e343baed7d99" path="/var/lib/kubelet/pods/d029bf9f-23a3-47c7-8eca-e343baed7d99/volumes" Nov 03 02:31:10.506708 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:10.506588 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:11.379552 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:11.379513 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:13.009750 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:31:13.009714 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:31:14.926551 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:14.926496 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg" podUID="29bbc3e3-3107-45c7-bd50-ee262c18c015" containerName="payload-pre-processing" containerID="cri-o://3c37d6bf12602f5f80d8a9504f1a05ae362b91e6f8327c66add47598988465e0" gracePeriod=25 Nov 03 02:31:15.214857 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:15.214832 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg" Nov 03 02:31:15.222035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:15.222010 5463 generic.go:356] "Generic (PLEG): container finished" podID="29bbc3e3-3107-45c7-bd50-ee262c18c015" containerID="3c37d6bf12602f5f80d8a9504f1a05ae362b91e6f8327c66add47598988465e0" exitCode=0 Nov 03 02:31:15.222151 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:15.222050 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg" Nov 03 02:31:15.222151 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:15.222092 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg" event={"ID":"29bbc3e3-3107-45c7-bd50-ee262c18c015","Type":"ContainerDied","Data":"3c37d6bf12602f5f80d8a9504f1a05ae362b91e6f8327c66add47598988465e0"} Nov 03 02:31:15.222151 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:15.222144 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg" event={"ID":"29bbc3e3-3107-45c7-bd50-ee262c18c015","Type":"ContainerDied","Data":"b066bd3dafba8d222ae83479126f2f97bd87d82d6c01f32b31b2b9fcdb30eff1"} Nov 03 02:31:15.222310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:15.222157 5463 scope.go:122] "RemoveContainer" containerID="3c37d6bf12602f5f80d8a9504f1a05ae362b91e6f8327c66add47598988465e0" Nov 03 02:31:15.231752 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:15.231732 5463 scope.go:122] "RemoveContainer" containerID="3c37d6bf12602f5f80d8a9504f1a05ae362b91e6f8327c66add47598988465e0" Nov 03 02:31:15.232030 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:31:15.232009 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c37d6bf12602f5f80d8a9504f1a05ae362b91e6f8327c66add47598988465e0\": container with ID starting with 3c37d6bf12602f5f80d8a9504f1a05ae362b91e6f8327c66add47598988465e0 not found: ID does not exist" containerID="3c37d6bf12602f5f80d8a9504f1a05ae362b91e6f8327c66add47598988465e0" Nov 03 02:31:15.232104 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:15.232043 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c37d6bf12602f5f80d8a9504f1a05ae362b91e6f8327c66add47598988465e0"} err="failed to get container status \"3c37d6bf12602f5f80d8a9504f1a05ae362b91e6f8327c66add47598988465e0\": rpc error: code = NotFound desc = could not find container \"3c37d6bf12602f5f80d8a9504f1a05ae362b91e6f8327c66add47598988465e0\": container with ID starting with 3c37d6bf12602f5f80d8a9504f1a05ae362b91e6f8327c66add47598988465e0 not found: ID does not exist" Nov 03 02:31:15.321934 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:15.321904 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29bbc3e3-3107-45c7-bd50-ee262c18c015-kube-api-access-vnr4f\" (UniqueName: \"kubernetes.io/projected/29bbc3e3-3107-45c7-bd50-ee262c18c015-kube-api-access-vnr4f\") pod \"29bbc3e3-3107-45c7-bd50-ee262c18c015\" (UID: \"29bbc3e3-3107-45c7-bd50-ee262c18c015\") " Nov 03 02:31:15.322065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:15.322032 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29bbc3e3-3107-45c7-bd50-ee262c18c015-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29bbc3e3-3107-45c7-bd50-ee262c18c015-plugins-config-volume\") pod \"29bbc3e3-3107-45c7-bd50-ee262c18c015\" (UID: \"29bbc3e3-3107-45c7-bd50-ee262c18c015\") " Nov 03 02:31:15.322389 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:15.322366 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29bbc3e3-3107-45c7-bd50-ee262c18c015-plugins-config-volume" pod "29bbc3e3-3107-45c7-bd50-ee262c18c015" (UID: "29bbc3e3-3107-45c7-bd50-ee262c18c015"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:31:15.322581 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:15.322563 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29bbc3e3-3107-45c7-bd50-ee262c18c015-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:15.324369 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:15.324333 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29bbc3e3-3107-45c7-bd50-ee262c18c015-kube-api-access-vnr4f" pod "29bbc3e3-3107-45c7-bd50-ee262c18c015" (UID: "29bbc3e3-3107-45c7-bd50-ee262c18c015"). InnerVolumeSpecName "kube-api-access-vnr4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:15.424249 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:15.424218 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnr4f\" (UniqueName: \"kubernetes.io/projected/29bbc3e3-3107-45c7-bd50-ee262c18c015-kube-api-access-vnr4f\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:15.643517 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:15.643489 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg"] Nov 03 02:31:15.646551 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:15.646523 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-496177-84785bddcf-89rqg"] Nov 03 02:31:16.044411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:16.044375 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:16.444638 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:16.444594 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29bbc3e3-3107-45c7-bd50-ee262c18c015" path="/var/lib/kubelet/pods/29bbc3e3-3107-45c7-bd50-ee262c18c015/volumes" Nov 03 02:31:18.153692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:18.153652 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:19.938999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:19.938961 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:21.600347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:21.600298 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:21.856937 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:21.856856 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:23.024032 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:23.024002 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:23.249531 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:23.249498 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:23.746316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:23.746281 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:25.583114 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583080 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz"] Nov 03 02:31:25.583678 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583664 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a7d39617-aba6-4951-b3ce-14f544abbf7a" containerName="istio-proxy" Nov 03 02:31:25.583720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583678 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d39617-aba6-4951-b3ce-14f544abbf7a" containerName="istio-proxy" Nov 03 02:31:25.583720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583700 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a7d39617-aba6-4951-b3ce-14f544abbf7a" containerName="istio-proxy" Nov 03 02:31:25.583720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583706 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d39617-aba6-4951-b3ce-14f544abbf7a" containerName="istio-proxy" Nov 03 02:31:25.583720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583716 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d029bf9f-23a3-47c7-8eca-e343baed7d99" containerName="istio-proxy" Nov 03 02:31:25.583870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583721 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d029bf9f-23a3-47c7-8eca-e343baed7d99" containerName="istio-proxy" Nov 03 02:31:25.583870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583730 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d029bf9f-23a3-47c7-8eca-e343baed7d99" containerName="istio-proxy" Nov 03 02:31:25.583870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583735 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d029bf9f-23a3-47c7-8eca-e343baed7d99" containerName="istio-proxy" Nov 03 02:31:25.583870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583745 5463 cpu_manager.go:414] "containerMap: removing container" podUID="29bbc3e3-3107-45c7-bd50-ee262c18c015" containerName="payload-pre-processing" Nov 03 02:31:25.583870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583750 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29bbc3e3-3107-45c7-bd50-ee262c18c015" containerName="payload-pre-processing" Nov 03 02:31:25.583870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583776 5463 cpu_manager.go:414] "containerMap: removing container" podUID="5d814d4b-c428-47d5-a594-578ba0b6bfd9" containerName="maas-api" Nov 03 02:31:25.583870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583780 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d814d4b-c428-47d5-a594-578ba0b6bfd9" containerName="maas-api" Nov 03 02:31:25.583870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583788 5463 cpu_manager.go:414] "containerMap: removing container" podUID="284b69a9-1ced-4c2f-9698-7682e921ca69" containerName="payload-pre-processing" Nov 03 02:31:25.583870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583792 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="284b69a9-1ced-4c2f-9698-7682e921ca69" containerName="payload-pre-processing" Nov 03 02:31:25.583870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583870 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d814d4b-c428-47d5-a594-578ba0b6bfd9" containerName="maas-api" Nov 03 02:31:25.584174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583880 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="284b69a9-1ced-4c2f-9698-7682e921ca69" containerName="payload-pre-processing" Nov 03 02:31:25.584174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583888 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d39617-aba6-4951-b3ce-14f544abbf7a" containerName="istio-proxy" Nov 03 02:31:25.584174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583895 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d39617-aba6-4951-b3ce-14f544abbf7a" containerName="istio-proxy" Nov 03 02:31:25.584174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583903 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d029bf9f-23a3-47c7-8eca-e343baed7d99" containerName="istio-proxy" Nov 03 02:31:25.584174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583909 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d029bf9f-23a3-47c7-8eca-e343baed7d99" containerName="istio-proxy" Nov 03 02:31:25.584174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.583914 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="29bbc3e3-3107-45c7-bd50-ee262c18c015" containerName="payload-pre-processing" Nov 03 02:31:25.586671 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.586654 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.605732 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.605709 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.605849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.605747 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.605894 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.605863 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.605950 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.605902 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkgvj\" (UniqueName: \"kubernetes.io/projected/21beedd4-fac4-4b92-9cf8-c2acb85422f4-kube-api-access-xkgvj\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.606009 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.605979 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.606064 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.606023 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.606229 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.606182 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.606360 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.606280 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.606360 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.606311 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.633970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.633940 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4"] Nov 03 02:31:25.637023 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.637003 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.686570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.686536 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz"] Nov 03 02:31:25.708184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.708137 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.708455 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.708266 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.708455 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.708420 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.708609 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.708472 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1bab284-6e6d-4ace-a4af-65fd25483815-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.708769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.708740 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.708887 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.708794 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkgvj\" (UniqueName: \"kubernetes.io/projected/21beedd4-fac4-4b92-9cf8-c2acb85422f4-kube-api-access-xkgvj\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.708887 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.708841 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.708887 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.708871 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.709057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.708922 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.709057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.708927 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.709057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.708946 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.709057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.708980 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.709057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.709019 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.709411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.709056 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.709411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.709103 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.709411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.709152 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.709411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.709216 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.709411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.709249 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.709411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.709311 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jklgb\" (UniqueName: \"kubernetes.io/projected/e1bab284-6e6d-4ace-a4af-65fd25483815-kube-api-access-jklgb\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.710053 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.710030 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.711135 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.711110 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.711308 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.711233 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.711378 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.711322 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.712475 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.712427 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.718948 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.718913 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.732671 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.732643 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4"] Nov 03 02:31:25.764984 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.764963 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.765552 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.765528 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkgvj\" (UniqueName: \"kubernetes.io/projected/21beedd4-fac4-4b92-9cf8-c2acb85422f4-kube-api-access-xkgvj\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.809948 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.809922 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.810085 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.809979 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.810085 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.810016 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.810085 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.810053 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.810394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.810091 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.810394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.810141 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.810394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.810282 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jklgb\" (UniqueName: \"kubernetes.io/projected/e1bab284-6e6d-4ace-a4af-65fd25483815-kube-api-access-jklgb\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.810394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.810328 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.810394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.810355 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1bab284-6e6d-4ace-a4af-65fd25483815-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.811182 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.811131 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.811182 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.811152 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.811182 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.811178 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1bab284-6e6d-4ace-a4af-65fd25483815-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.811182 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.811216 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.811500 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.811325 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.812687 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.812502 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.814825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.814801 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.867646 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.867578 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.867769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.867754 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jklgb\" (UniqueName: \"kubernetes.io/projected/e1bab284-6e6d-4ace-a4af-65fd25483815-kube-api-access-jklgb\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:25.899688 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.899661 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:25.949821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:25.949790 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:26.069493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:26.069462 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:26.317333 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:31:26.317301 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21beedd4_fac4_4b92_9cf8_c2acb85422f4.slice/crio-b92bbaad9462c7ac79c520891381f7fdb4d5ba7f077f69b89599d94f688fd8bb WatchSource:0}: Error finding container b92bbaad9462c7ac79c520891381f7fdb4d5ba7f077f69b89599d94f688fd8bb: Status 404 returned error can't find the container with id b92bbaad9462c7ac79c520891381f7fdb4d5ba7f077f69b89599d94f688fd8bb Nov 03 02:31:26.319369 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:26.319324 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:31:26.319465 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:26.319415 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:31:26.319465 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:26.319455 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:31:26.320443 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:26.320416 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz"] Nov 03 02:31:26.365128 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:31:26.365101 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1bab284_6e6d_4ace_a4af_65fd25483815.slice/crio-bd41caa59ef750ecf1ea324857f18d5bc5e60f0f15c15fa18267ce51b2b2e458 WatchSource:0}: Error finding container bd41caa59ef750ecf1ea324857f18d5bc5e60f0f15c15fa18267ce51b2b2e458: Status 404 returned error can't find the container with id bd41caa59ef750ecf1ea324857f18d5bc5e60f0f15c15fa18267ce51b2b2e458 Nov 03 02:31:26.367866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:26.367830 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:31:26.367953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:26.367901 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:31:26.367953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:26.367853 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4"] Nov 03 02:31:26.368022 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:26.367934 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:31:26.590409 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:26.590319 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh"] Nov 03 02:31:26.696023 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:26.695986 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv"] Nov 03 02:31:26.999710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:26.996545 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:27.173217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:27.173059 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:27.332064 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:27.331976 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" event={"ID":"e1bab284-6e6d-4ace-a4af-65fd25483815","Type":"ContainerStarted","Data":"3556c9e31dd5ed5f894b45a5d03560a1792546d9015fbdfc2d236e66fa274c22"} Nov 03 02:31:27.332064 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:27.332033 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" event={"ID":"e1bab284-6e6d-4ace-a4af-65fd25483815","Type":"ContainerStarted","Data":"bd41caa59ef750ecf1ea324857f18d5bc5e60f0f15c15fa18267ce51b2b2e458"} Nov 03 02:31:27.334078 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:27.334053 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" event={"ID":"21beedd4-fac4-4b92-9cf8-c2acb85422f4","Type":"ContainerStarted","Data":"ec2f75958173a8cfbb9f5920d5000fa0da79d56ef2b1f8be4d68a9c7b8b08a48"} Nov 03 02:31:27.334212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:27.334101 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" event={"ID":"21beedd4-fac4-4b92-9cf8-c2acb85422f4","Type":"ContainerStarted","Data":"b92bbaad9462c7ac79c520891381f7fdb4d5ba7f077f69b89599d94f688fd8bb"} Nov 03 02:31:27.465217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:27.458684 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" podStartSLOduration=2.458663484 podStartE2EDuration="2.458663484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:31:27.456551233 +0000 UTC m=+2369.140660389" watchObservedRunningTime="2026-11-03 02:31:27.458663484 +0000 UTC m=+2369.142772637" Nov 03 02:31:27.599308 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:27.598391 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" podStartSLOduration=2.598369793 podStartE2EDuration="2.598369793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:31:27.581849085 +0000 UTC m=+2369.265958238" watchObservedRunningTime="2026-11-03 02:31:27.598369793 +0000 UTC m=+2369.282478952" Nov 03 02:31:27.904854 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:27.904493 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:27.905664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:27.905115 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 02:31:27.905664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:27.905171 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" podUID="21beedd4-fac4-4b92-9cf8-c2acb85422f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 02:31:27.959839 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:27.959807 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:27.960123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:27.960102 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 03 02:31:27.960326 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:27.960302 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" podUID="e1bab284-6e6d-4ace-a4af-65fd25483815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 03 02:31:28.903216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:28.902383 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 02:31:28.903216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:28.902456 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" podUID="21beedd4-fac4-4b92-9cf8-c2acb85422f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 02:31:28.961218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:28.951497 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 03 02:31:28.961218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:28.951553 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" podUID="e1bab284-6e6d-4ace-a4af-65fd25483815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 03 02:31:29.525261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:29.525229 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:29.909043 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:29.909001 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 02:31:29.909698 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:29.909661 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" podUID="21beedd4-fac4-4b92-9cf8-c2acb85422f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 02:31:29.991216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:29.990652 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 03 02:31:29.991216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:29.990760 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" podUID="e1bab284-6e6d-4ace-a4af-65fd25483815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 03 02:31:30.006217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.004807 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s"] Nov 03 02:31:30.006217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.005070 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s" podUID="2063d869-2397-4105-bdb3-ea049d4bd196" containerName="payload-processing" containerID="cri-o://5835b707361de2750c95a27b6f3e448f169270632f5506bf49a705b58ea0ccc4" gracePeriod=30 Nov 03 02:31:30.055547 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.047179 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq"] Nov 03 02:31:30.078441 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.078401 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq" podUID="dad5987b-deb7-4e6a-b8a5-8e2821700969" containerName="payload-pre-processing" containerID="cri-o://953dff0687d13f8a9d209bbd05c5a3c2b46cddbf3fb7aa46e618463c9db63f9c" gracePeriod=30 Nov 03 02:31:30.133991 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.133876 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:30.291801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.291767 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:30.445794 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.445323 5463 generic.go:356] "Generic (PLEG): container finished" podID="dad5987b-deb7-4e6a-b8a5-8e2821700969" containerID="953dff0687d13f8a9d209bbd05c5a3c2b46cddbf3fb7aa46e618463c9db63f9c" exitCode=0 Nov 03 02:31:30.531006 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.530878 5463 generic.go:356] "Generic (PLEG): container finished" podID="2063d869-2397-4105-bdb3-ea049d4bd196" containerID="5835b707361de2750c95a27b6f3e448f169270632f5506bf49a705b58ea0ccc4" exitCode=0 Nov 03 02:31:30.531801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.531772 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq" event={"ID":"dad5987b-deb7-4e6a-b8a5-8e2821700969","Type":"ContainerDied","Data":"953dff0687d13f8a9d209bbd05c5a3c2b46cddbf3fb7aa46e618463c9db63f9c"} Nov 03 02:31:30.531934 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.531839 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s" event={"ID":"2063d869-2397-4105-bdb3-ea049d4bd196","Type":"ContainerDied","Data":"5835b707361de2750c95a27b6f3e448f169270632f5506bf49a705b58ea0ccc4"} Nov 03 02:31:30.628680 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.628644 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s" Nov 03 02:31:30.771215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.770575 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2063d869-2397-4105-bdb3-ea049d4bd196-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2063d869-2397-4105-bdb3-ea049d4bd196-plugins-config-volume\") pod \"2063d869-2397-4105-bdb3-ea049d4bd196\" (UID: \"2063d869-2397-4105-bdb3-ea049d4bd196\") " Nov 03 02:31:30.771215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.770777 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2063d869-2397-4105-bdb3-ea049d4bd196-kube-api-access-q6xd6\" (UniqueName: \"kubernetes.io/projected/2063d869-2397-4105-bdb3-ea049d4bd196-kube-api-access-q6xd6\") pod \"2063d869-2397-4105-bdb3-ea049d4bd196\" (UID: \"2063d869-2397-4105-bdb3-ea049d4bd196\") " Nov 03 02:31:30.845438 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.830138 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2063d869-2397-4105-bdb3-ea049d4bd196-plugins-config-volume" pod "2063d869-2397-4105-bdb3-ea049d4bd196" (UID: "2063d869-2397-4105-bdb3-ea049d4bd196"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:31:30.845438 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.844374 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2063d869-2397-4105-bdb3-ea049d4bd196-kube-api-access-q6xd6" pod "2063d869-2397-4105-bdb3-ea049d4bd196" (UID: "2063d869-2397-4105-bdb3-ea049d4bd196"). InnerVolumeSpecName "kube-api-access-q6xd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:30.901403 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.901278 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2063d869-2397-4105-bdb3-ea049d4bd196-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:30.901403 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.901313 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6xd6\" (UniqueName: \"kubernetes.io/projected/2063d869-2397-4105-bdb3-ea049d4bd196-kube-api-access-q6xd6\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:30.902271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.902245 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 02:31:30.902449 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.902424 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" podUID="21beedd4-fac4-4b92-9cf8-c2acb85422f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 02:31:30.939285 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.939251 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq" Nov 03 02:31:30.958209 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.955250 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 03 02:31:30.958209 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:30.955322 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" podUID="e1bab284-6e6d-4ace-a4af-65fd25483815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 03 02:31:31.114262 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.109750 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad5987b-deb7-4e6a-b8a5-8e2821700969-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad5987b-deb7-4e6a-b8a5-8e2821700969-plugins-config-volume\") pod \"dad5987b-deb7-4e6a-b8a5-8e2821700969\" (UID: \"dad5987b-deb7-4e6a-b8a5-8e2821700969\") " Nov 03 02:31:31.114262 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.114065 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad5987b-deb7-4e6a-b8a5-8e2821700969-kube-api-access-smz6l\" (UniqueName: \"kubernetes.io/projected/dad5987b-deb7-4e6a-b8a5-8e2821700969-kube-api-access-smz6l\") pod \"dad5987b-deb7-4e6a-b8a5-8e2821700969\" (UID: \"dad5987b-deb7-4e6a-b8a5-8e2821700969\") " Nov 03 02:31:31.114262 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.113496 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad5987b-deb7-4e6a-b8a5-8e2821700969-plugins-config-volume" pod "dad5987b-deb7-4e6a-b8a5-8e2821700969" (UID: "dad5987b-deb7-4e6a-b8a5-8e2821700969"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:31:31.114671 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.114647 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad5987b-deb7-4e6a-b8a5-8e2821700969-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:31.116812 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.116788 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad5987b-deb7-4e6a-b8a5-8e2821700969-kube-api-access-smz6l" pod "dad5987b-deb7-4e6a-b8a5-8e2821700969" (UID: "dad5987b-deb7-4e6a-b8a5-8e2821700969"). InnerVolumeSpecName "kube-api-access-smz6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:31.216358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.216325 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smz6l\" (UniqueName: \"kubernetes.io/projected/dad5987b-deb7-4e6a-b8a5-8e2821700969-kube-api-access-smz6l\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:31.540764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.540732 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq" Nov 03 02:31:31.540764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.540754 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq" event={"ID":"dad5987b-deb7-4e6a-b8a5-8e2821700969","Type":"ContainerDied","Data":"b710546e8e342765c3d338c7e26a8dcd77af736a27f6a811b0811ffa0187e221"} Nov 03 02:31:31.541026 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.540870 5463 scope.go:122] "RemoveContainer" containerID="953dff0687d13f8a9d209bbd05c5a3c2b46cddbf3fb7aa46e618463c9db63f9c" Nov 03 02:31:31.542537 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.542504 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s" event={"ID":"2063d869-2397-4105-bdb3-ea049d4bd196","Type":"ContainerDied","Data":"d1277fdcdd3e2b1a2bf258bdf969aa73e778771e6dd0ba81e63fe8e9d5e720b9"} Nov 03 02:31:31.542706 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.542576 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s" Nov 03 02:31:31.593149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.593022 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh" podUID="2e7a3f30-dace-4684-9257-b2b103a2af17" containerName="payload-processing" containerID="cri-o://0b7ebdd6bdf2b63c2ba3fde0911e5c69dbb0536cd3e96d384b17a275690b0e95" gracePeriod=25 Nov 03 02:31:31.608928 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.608904 5463 scope.go:122] "RemoveContainer" containerID="5835b707361de2750c95a27b6f3e448f169270632f5506bf49a705b58ea0ccc4" Nov 03 02:31:31.626608 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.626585 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:31.704213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.703076 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv" podUID="49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135" containerName="payload-pre-processing" containerID="cri-o://b1eaff6411e933bc3c157cdc6c483e5dc7d68babd83e694f8b62cc305296bbfd" gracePeriod=25 Nov 03 02:31:31.725179 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.725146 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq"] Nov 03 02:31:31.729837 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.729812 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8857cb-6fbd9996c5-w9qxq"] Nov 03 02:31:31.808446 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.808111 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:31.916335 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.916295 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 02:31:31.916524 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.916363 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" podUID="21beedd4-fac4-4b92-9cf8-c2acb85422f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 02:31:31.927238 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.927185 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s"] Nov 03 02:31:31.936054 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.936026 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8857cb-5d69867fdb-kfb8s"] Nov 03 02:31:31.953804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.953535 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 03 02:31:31.953804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:31.953591 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" podUID="e1bab284-6e6d-4ace-a4af-65fd25483815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 03 02:31:32.138473 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.138449 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh" Nov 03 02:31:32.230049 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.230013 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e7a3f30-dace-4684-9257-b2b103a2af17-kube-api-access-trdrx\" (UniqueName: \"kubernetes.io/projected/2e7a3f30-dace-4684-9257-b2b103a2af17-kube-api-access-trdrx\") pod \"2e7a3f30-dace-4684-9257-b2b103a2af17\" (UID: \"2e7a3f30-dace-4684-9257-b2b103a2af17\") " Nov 03 02:31:32.230233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.230076 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e7a3f30-dace-4684-9257-b2b103a2af17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e7a3f30-dace-4684-9257-b2b103a2af17-plugins-config-volume\") pod \"2e7a3f30-dace-4684-9257-b2b103a2af17\" (UID: \"2e7a3f30-dace-4684-9257-b2b103a2af17\") " Nov 03 02:31:32.231042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.231013 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7a3f30-dace-4684-9257-b2b103a2af17-plugins-config-volume" pod "2e7a3f30-dace-4684-9257-b2b103a2af17" (UID: "2e7a3f30-dace-4684-9257-b2b103a2af17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:31:32.233052 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.232984 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7a3f30-dace-4684-9257-b2b103a2af17-kube-api-access-trdrx" pod "2e7a3f30-dace-4684-9257-b2b103a2af17" (UID: "2e7a3f30-dace-4684-9257-b2b103a2af17"). InnerVolumeSpecName "kube-api-access-trdrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:32.317859 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.317814 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv" Nov 03 02:31:32.331619 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.331589 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trdrx\" (UniqueName: \"kubernetes.io/projected/2e7a3f30-dace-4684-9257-b2b103a2af17-kube-api-access-trdrx\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:32.331619 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.331617 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e7a3f30-dace-4684-9257-b2b103a2af17-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:32.432917 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.432882 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135-plugins-config-volume\") pod \"49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135\" (UID: \"49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135\") " Nov 03 02:31:32.433081 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.432999 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135-kube-api-access-vdcw9\" (UniqueName: \"kubernetes.io/projected/49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135-kube-api-access-vdcw9\") pod \"49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135\" (UID: \"49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135\") " Nov 03 02:31:32.433444 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.433416 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135-plugins-config-volume" pod "49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135" (UID: "49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:31:32.433679 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.433659 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:32.435217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.435177 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135-kube-api-access-vdcw9" pod "49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135" (UID: "49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135"). InnerVolumeSpecName "kube-api-access-vdcw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:32.483206 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.483131 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2063d869-2397-4105-bdb3-ea049d4bd196" path="/var/lib/kubelet/pods/2063d869-2397-4105-bdb3-ea049d4bd196/volumes" Nov 03 02:31:32.483786 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.483768 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad5987b-deb7-4e6a-b8a5-8e2821700969" path="/var/lib/kubelet/pods/dad5987b-deb7-4e6a-b8a5-8e2821700969/volumes" Nov 03 02:31:32.535275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.535235 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdcw9\" (UniqueName: \"kubernetes.io/projected/49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135-kube-api-access-vdcw9\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:32.556900 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.556859 5463 generic.go:356] "Generic (PLEG): container finished" podID="2e7a3f30-dace-4684-9257-b2b103a2af17" containerID="0b7ebdd6bdf2b63c2ba3fde0911e5c69dbb0536cd3e96d384b17a275690b0e95" exitCode=0 Nov 03 02:31:32.557123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.556983 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh" Nov 03 02:31:32.557123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.557012 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh" event={"ID":"2e7a3f30-dace-4684-9257-b2b103a2af17","Type":"ContainerDied","Data":"0b7ebdd6bdf2b63c2ba3fde0911e5c69dbb0536cd3e96d384b17a275690b0e95"} Nov 03 02:31:32.557123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.557062 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh" event={"ID":"2e7a3f30-dace-4684-9257-b2b103a2af17","Type":"ContainerDied","Data":"079b63befe50e2c39f58629e98589abc296fe91e19846be61e7656a7480a954e"} Nov 03 02:31:32.557123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.557094 5463 scope.go:122] "RemoveContainer" containerID="0b7ebdd6bdf2b63c2ba3fde0911e5c69dbb0536cd3e96d384b17a275690b0e95" Nov 03 02:31:32.565758 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.565725 5463 generic.go:356] "Generic (PLEG): container finished" podID="49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135" containerID="b1eaff6411e933bc3c157cdc6c483e5dc7d68babd83e694f8b62cc305296bbfd" exitCode=0 Nov 03 02:31:32.565971 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.565806 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv" event={"ID":"49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135","Type":"ContainerDied","Data":"b1eaff6411e933bc3c157cdc6c483e5dc7d68babd83e694f8b62cc305296bbfd"} Nov 03 02:31:32.565971 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.565843 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv" Nov 03 02:31:32.565971 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.565865 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv" event={"ID":"49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135","Type":"ContainerDied","Data":"2958554582e6f8482fe39034b4a18a52274651c6ecb76e007485419f3623ddfe"} Nov 03 02:31:32.579043 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.579012 5463 scope.go:122] "RemoveContainer" containerID="0b7ebdd6bdf2b63c2ba3fde0911e5c69dbb0536cd3e96d384b17a275690b0e95" Nov 03 02:31:32.579456 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:31:32.579429 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7ebdd6bdf2b63c2ba3fde0911e5c69dbb0536cd3e96d384b17a275690b0e95\": container with ID starting with 0b7ebdd6bdf2b63c2ba3fde0911e5c69dbb0536cd3e96d384b17a275690b0e95 not found: ID does not exist" containerID="0b7ebdd6bdf2b63c2ba3fde0911e5c69dbb0536cd3e96d384b17a275690b0e95" Nov 03 02:31:32.579555 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.579467 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7ebdd6bdf2b63c2ba3fde0911e5c69dbb0536cd3e96d384b17a275690b0e95"} err="failed to get container status \"0b7ebdd6bdf2b63c2ba3fde0911e5c69dbb0536cd3e96d384b17a275690b0e95\": rpc error: code = NotFound desc = could not find container \"0b7ebdd6bdf2b63c2ba3fde0911e5c69dbb0536cd3e96d384b17a275690b0e95\": container with ID starting with 0b7ebdd6bdf2b63c2ba3fde0911e5c69dbb0536cd3e96d384b17a275690b0e95 not found: ID does not exist" Nov 03 02:31:32.579555 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.579493 5463 scope.go:122] "RemoveContainer" containerID="b1eaff6411e933bc3c157cdc6c483e5dc7d68babd83e694f8b62cc305296bbfd" Nov 03 02:31:32.581434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.581405 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:32.591079 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.591057 5463 scope.go:122] "RemoveContainer" containerID="b1eaff6411e933bc3c157cdc6c483e5dc7d68babd83e694f8b62cc305296bbfd" Nov 03 02:31:32.591485 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:31:32.591461 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1eaff6411e933bc3c157cdc6c483e5dc7d68babd83e694f8b62cc305296bbfd\": container with ID starting with b1eaff6411e933bc3c157cdc6c483e5dc7d68babd83e694f8b62cc305296bbfd not found: ID does not exist" containerID="b1eaff6411e933bc3c157cdc6c483e5dc7d68babd83e694f8b62cc305296bbfd" Nov 03 02:31:32.591569 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.591493 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1eaff6411e933bc3c157cdc6c483e5dc7d68babd83e694f8b62cc305296bbfd"} err="failed to get container status \"b1eaff6411e933bc3c157cdc6c483e5dc7d68babd83e694f8b62cc305296bbfd\": rpc error: code = NotFound desc = could not find container \"b1eaff6411e933bc3c157cdc6c483e5dc7d68babd83e694f8b62cc305296bbfd\": container with ID starting with b1eaff6411e933bc3c157cdc6c483e5dc7d68babd83e694f8b62cc305296bbfd not found: ID does not exist" Nov 03 02:31:32.726163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.726088 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv"] Nov 03 02:31:32.731452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.731424 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7f760c-85f7dcfb55-wp9dv"] Nov 03 02:31:32.840721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.840688 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh"] Nov 03 02:31:32.843388 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.843365 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7f760c-95f5b87cc-trznh"] Nov 03 02:31:32.904461 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.904435 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:32.904797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.904779 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:32.905564 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.905537 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:31:32.954858 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.954833 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:32.955216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.955168 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:32.955710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:32.955694 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:31:33.127149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:33.127070 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:33.574767 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:33.574732 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2"] Nov 03 02:31:33.575012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:33.574987 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" podUID="17c81363-4f1f-4550-ba15-35f9e86eccf7" containerName="istio-proxy" containerID="cri-o://da4f47d76050ddbe3590a7f86e4ea9ea6c52cdfb7ae19d25894feaf30463371f" gracePeriod=30 Nov 03 02:31:33.662387 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:33.662355 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:34.444442 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:34.444406 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e7a3f30-dace-4684-9257-b2b103a2af17" path="/var/lib/kubelet/pods/2e7a3f30-dace-4684-9257-b2b103a2af17/volumes" Nov 03 02:31:34.444963 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:34.444944 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135" path="/var/lib/kubelet/pods/49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135/volumes" Nov 03 02:31:35.880027 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:35.879987 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:31:35.880475 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:35.880060 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" podUID="17c81363-4f1f-4550-ba15-35f9e86eccf7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:31:37.034766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:37.034727 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:38.857067 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.857009 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:31:38.916429 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.916401 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-workload-socket\") pod \"17c81363-4f1f-4550-ba15-35f9e86eccf7\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " Nov 03 02:31:38.916562 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.916460 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-data\") pod \"17c81363-4f1f-4550-ba15-35f9e86eccf7\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " Nov 03 02:31:38.916562 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.916507 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-workload-certs\") pod \"17c81363-4f1f-4550-ba15-35f9e86eccf7\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " Nov 03 02:31:38.916562 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.916530 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-podinfo\") pod \"17c81363-4f1f-4550-ba15-35f9e86eccf7\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " Nov 03 02:31:38.916755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.916582 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-token\" (UniqueName: \"kubernetes.io/projected/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-token\") pod \"17c81363-4f1f-4550-ba15-35f9e86eccf7\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " Nov 03 02:31:38.916755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.916619 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17c81363-4f1f-4550-ba15-35f9e86eccf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c81363-4f1f-4550-ba15-35f9e86eccf7-istiod-ca-cert\") pod \"17c81363-4f1f-4550-ba15-35f9e86eccf7\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " Nov 03 02:31:38.916755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.916638 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-workload-socket" pod "17c81363-4f1f-4550-ba15-35f9e86eccf7" (UID: "17c81363-4f1f-4550-ba15-35f9e86eccf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:31:38.916755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.916680 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-envoy\") pod \"17c81363-4f1f-4550-ba15-35f9e86eccf7\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " Nov 03 02:31:38.916755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.916738 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-credential-socket\") pod \"17c81363-4f1f-4550-ba15-35f9e86eccf7\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " Nov 03 02:31:38.917113 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.916750 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-data" pod "17c81363-4f1f-4550-ba15-35f9e86eccf7" (UID: "17c81363-4f1f-4550-ba15-35f9e86eccf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:31:38.917113 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.916791 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c81363-4f1f-4550-ba15-35f9e86eccf7-kube-api-access-5zw94\" (UniqueName: \"kubernetes.io/projected/17c81363-4f1f-4550-ba15-35f9e86eccf7-kube-api-access-5zw94\") pod \"17c81363-4f1f-4550-ba15-35f9e86eccf7\" (UID: \"17c81363-4f1f-4550-ba15-35f9e86eccf7\") " Nov 03 02:31:38.917113 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.916791 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-workload-certs" pod "17c81363-4f1f-4550-ba15-35f9e86eccf7" (UID: "17c81363-4f1f-4550-ba15-35f9e86eccf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:31:38.917113 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.917045 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c81363-4f1f-4550-ba15-35f9e86eccf7-istiod-ca-cert" pod "17c81363-4f1f-4550-ba15-35f9e86eccf7" (UID: "17c81363-4f1f-4550-ba15-35f9e86eccf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:31:38.917457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.917258 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-credential-socket" pod "17c81363-4f1f-4550-ba15-35f9e86eccf7" (UID: "17c81363-4f1f-4550-ba15-35f9e86eccf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:31:38.917574 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.917547 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:38.917663 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.917574 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:38.917663 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.917589 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:38.917663 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.917604 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c81363-4f1f-4550-ba15-35f9e86eccf7-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:38.917663 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.917619 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:38.919067 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.919041 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-envoy" pod "17c81363-4f1f-4550-ba15-35f9e86eccf7" (UID: "17c81363-4f1f-4550-ba15-35f9e86eccf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:31:38.919168 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.919124 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c81363-4f1f-4550-ba15-35f9e86eccf7-kube-api-access-5zw94" pod "17c81363-4f1f-4550-ba15-35f9e86eccf7" (UID: "17c81363-4f1f-4550-ba15-35f9e86eccf7"). InnerVolumeSpecName "kube-api-access-5zw94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:38.919168 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.919158 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-token" pod "17c81363-4f1f-4550-ba15-35f9e86eccf7" (UID: "17c81363-4f1f-4550-ba15-35f9e86eccf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:38.919335 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:38.919319 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-podinfo" pod "17c81363-4f1f-4550-ba15-35f9e86eccf7" (UID: "17c81363-4f1f-4550-ba15-35f9e86eccf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:31:39.019342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:39.019309 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:39.019342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:39.019344 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:39.019582 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:39.019360 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c81363-4f1f-4550-ba15-35f9e86eccf7-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:39.019582 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:39.019373 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zw94\" (UniqueName: \"kubernetes.io/projected/17c81363-4f1f-4550-ba15-35f9e86eccf7-kube-api-access-5zw94\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:39.625014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:39.624979 5463 generic.go:356] "Generic (PLEG): container finished" podID="17c81363-4f1f-4550-ba15-35f9e86eccf7" containerID="da4f47d76050ddbe3590a7f86e4ea9ea6c52cdfb7ae19d25894feaf30463371f" exitCode=0 Nov 03 02:31:39.625252 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:39.625036 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" Nov 03 02:31:39.625252 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:39.625061 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" event={"ID":"17c81363-4f1f-4550-ba15-35f9e86eccf7","Type":"ContainerDied","Data":"da4f47d76050ddbe3590a7f86e4ea9ea6c52cdfb7ae19d25894feaf30463371f"} Nov 03 02:31:39.625252 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:39.625131 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2" event={"ID":"17c81363-4f1f-4550-ba15-35f9e86eccf7","Type":"ContainerDied","Data":"e843ed797c815fe597795684f2a2b5c6594c4f257527aef0707777905b7fa834"} Nov 03 02:31:39.625252 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:39.625162 5463 scope.go:122] "RemoveContainer" containerID="da4f47d76050ddbe3590a7f86e4ea9ea6c52cdfb7ae19d25894feaf30463371f" Nov 03 02:31:39.635723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:39.635697 5463 scope.go:122] "RemoveContainer" containerID="84400737403c98c63902b2439174cd25ae46e5336a60892423e2b597ebf71be0" Nov 03 02:31:39.645115 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:39.645100 5463 scope.go:122] "RemoveContainer" containerID="da4f47d76050ddbe3590a7f86e4ea9ea6c52cdfb7ae19d25894feaf30463371f" Nov 03 02:31:39.645392 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:31:39.645371 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da4f47d76050ddbe3590a7f86e4ea9ea6c52cdfb7ae19d25894feaf30463371f\": container with ID starting with da4f47d76050ddbe3590a7f86e4ea9ea6c52cdfb7ae19d25894feaf30463371f not found: ID does not exist" containerID="da4f47d76050ddbe3590a7f86e4ea9ea6c52cdfb7ae19d25894feaf30463371f" Nov 03 02:31:39.645462 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:39.645401 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da4f47d76050ddbe3590a7f86e4ea9ea6c52cdfb7ae19d25894feaf30463371f"} err="failed to get container status \"da4f47d76050ddbe3590a7f86e4ea9ea6c52cdfb7ae19d25894feaf30463371f\": rpc error: code = NotFound desc = could not find container \"da4f47d76050ddbe3590a7f86e4ea9ea6c52cdfb7ae19d25894feaf30463371f\": container with ID starting with da4f47d76050ddbe3590a7f86e4ea9ea6c52cdfb7ae19d25894feaf30463371f not found: ID does not exist" Nov 03 02:31:39.645462 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:39.645417 5463 scope.go:122] "RemoveContainer" containerID="84400737403c98c63902b2439174cd25ae46e5336a60892423e2b597ebf71be0" Nov 03 02:31:39.645662 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:31:39.645645 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84400737403c98c63902b2439174cd25ae46e5336a60892423e2b597ebf71be0\": container with ID starting with 84400737403c98c63902b2439174cd25ae46e5336a60892423e2b597ebf71be0 not found: ID does not exist" containerID="84400737403c98c63902b2439174cd25ae46e5336a60892423e2b597ebf71be0" Nov 03 02:31:39.645710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:39.645667 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84400737403c98c63902b2439174cd25ae46e5336a60892423e2b597ebf71be0"} err="failed to get container status \"84400737403c98c63902b2439174cd25ae46e5336a60892423e2b597ebf71be0\": rpc error: code = NotFound desc = could not find container \"84400737403c98c63902b2439174cd25ae46e5336a60892423e2b597ebf71be0\": container with ID starting with 84400737403c98c63902b2439174cd25ae46e5336a60892423e2b597ebf71be0 not found: ID does not exist" Nov 03 02:31:39.791418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:39.791383 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2"] Nov 03 02:31:39.841593 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:39.841559 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8857cb-openshift-default-6bf54c69cf-nm4b2"] Nov 03 02:31:40.443139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.443099 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17c81363-4f1f-4550-ba15-35f9e86eccf7" path="/var/lib/kubelet/pods/17c81363-4f1f-4550-ba15-35f9e86eccf7/volumes" Nov 03 02:31:40.698506 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.698411 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2"] Nov 03 02:31:40.699075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.699058 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2e7a3f30-dace-4684-9257-b2b103a2af17" containerName="payload-processing" Nov 03 02:31:40.699138 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.699099 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e7a3f30-dace-4684-9257-b2b103a2af17" containerName="payload-processing" Nov 03 02:31:40.699138 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.699136 5463 cpu_manager.go:414] "containerMap: removing container" podUID="17c81363-4f1f-4550-ba15-35f9e86eccf7" containerName="istio-proxy" Nov 03 02:31:40.699273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.699142 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c81363-4f1f-4550-ba15-35f9e86eccf7" containerName="istio-proxy" Nov 03 02:31:40.699273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.699151 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2063d869-2397-4105-bdb3-ea049d4bd196" containerName="payload-processing" Nov 03 02:31:40.699273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.699159 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2063d869-2397-4105-bdb3-ea049d4bd196" containerName="payload-processing" Nov 03 02:31:40.699273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.699175 5463 cpu_manager.go:414] "containerMap: removing container" podUID="49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135" containerName="payload-pre-processing" Nov 03 02:31:40.699273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.699184 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135" containerName="payload-pre-processing" Nov 03 02:31:40.699273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.699215 5463 cpu_manager.go:414] "containerMap: removing container" podUID="17c81363-4f1f-4550-ba15-35f9e86eccf7" containerName="istio-proxy" Nov 03 02:31:40.699273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.699222 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c81363-4f1f-4550-ba15-35f9e86eccf7" containerName="istio-proxy" Nov 03 02:31:40.699273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.699242 5463 cpu_manager.go:414] "containerMap: removing container" podUID="dad5987b-deb7-4e6a-b8a5-8e2821700969" containerName="payload-pre-processing" Nov 03 02:31:40.699273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.699250 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad5987b-deb7-4e6a-b8a5-8e2821700969" containerName="payload-pre-processing" Nov 03 02:31:40.699642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.699380 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e7a3f30-dace-4684-9257-b2b103a2af17" containerName="payload-processing" Nov 03 02:31:40.699642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.699396 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c81363-4f1f-4550-ba15-35f9e86eccf7" containerName="istio-proxy" Nov 03 02:31:40.699642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.699403 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2063d869-2397-4105-bdb3-ea049d4bd196" containerName="payload-processing" Nov 03 02:31:40.699642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.699409 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a6408e-d9c9-4cf3-9c6f-80a3f2a6e135" containerName="payload-pre-processing" Nov 03 02:31:40.699642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.699419 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c81363-4f1f-4550-ba15-35f9e86eccf7" containerName="istio-proxy" Nov 03 02:31:40.699642 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.699430 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad5987b-deb7-4e6a-b8a5-8e2821700969" containerName="payload-pre-processing" Nov 03 02:31:40.702737 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.702718 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.739503 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.739478 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad2720fd-aa19-4213-a377-c98474134c93-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.739622 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.739506 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.739622 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.739545 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvz9q\" (UniqueName: \"kubernetes.io/projected/ad2720fd-aa19-4213-a377-c98474134c93-kube-api-access-kvz9q\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.739717 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.739656 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.739717 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.739699 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.739852 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.739832 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.739943 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.739924 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad2720fd-aa19-4213-a377-c98474134c93-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.740005 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.739956 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.740005 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.739988 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad2720fd-aa19-4213-a377-c98474134c93-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.809992 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.809954 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2"] Nov 03 02:31:40.841064 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.841035 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvz9q\" (UniqueName: \"kubernetes.io/projected/ad2720fd-aa19-4213-a377-c98474134c93-kube-api-access-kvz9q\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.841261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.841095 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.841261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.841125 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.841261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.841158 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.841261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.841234 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad2720fd-aa19-4213-a377-c98474134c93-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.841261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.841255 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.841573 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.841280 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad2720fd-aa19-4213-a377-c98474134c93-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.841573 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.841363 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad2720fd-aa19-4213-a377-c98474134c93-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.841573 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.841388 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.841728 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.841704 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.841833 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.841807 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.841902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.841830 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.841902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.841855 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.842083 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.842066 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad2720fd-aa19-4213-a377-c98474134c93-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.843536 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.843515 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.843780 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.843761 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad2720fd-aa19-4213-a377-c98474134c93-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.898364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.898337 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad2720fd-aa19-4213-a377-c98474134c93-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:40.898640 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:40.898624 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvz9q\" (UniqueName: \"kubernetes.io/projected/ad2720fd-aa19-4213-a377-c98474134c93-kube-api-access-kvz9q\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:41.015602 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:41.015526 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:41.435630 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:31:41.435600 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2720fd_aa19_4213_a377_c98474134c93.slice/crio-13637b00c35bf69e3292f37d2bd1797f4ef1f89630945de7688be9d3f0634e43 WatchSource:0}: Error finding container 13637b00c35bf69e3292f37d2bd1797f4ef1f89630945de7688be9d3f0634e43: Status 404 returned error can't find the container with id 13637b00c35bf69e3292f37d2bd1797f4ef1f89630945de7688be9d3f0634e43 Nov 03 02:31:41.438830 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:41.438791 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:31:41.438921 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:41.438865 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:31:41.438921 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:41.438895 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:31:41.439106 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:41.439083 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2"] Nov 03 02:31:41.640813 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:41.640780 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" event={"ID":"ad2720fd-aa19-4213-a377-c98474134c93","Type":"ContainerStarted","Data":"abdd6febda5525eb728fb19fa6869d93e56f230bfbb19cad71001345aed21110"} Nov 03 02:31:41.641175 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:41.640833 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" event={"ID":"ad2720fd-aa19-4213-a377-c98474134c93","Type":"ContainerStarted","Data":"13637b00c35bf69e3292f37d2bd1797f4ef1f89630945de7688be9d3f0634e43"} Nov 03 02:31:41.758999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:41.758890 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" podStartSLOduration=1.758871202 podStartE2EDuration="1.758871202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:31:41.757610653 +0000 UTC m=+2383.441719812" watchObservedRunningTime="2026-11-03 02:31:41.758871202 +0000 UTC m=+2383.442980358" Nov 03 02:31:42.016030 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:42.015950 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:42.017615 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:42.017587 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Nov 03 02:31:42.017722 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:42.017639 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" podUID="ad2720fd-aa19-4213-a377-c98474134c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Nov 03 02:31:42.225748 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:42.225687 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw"] Nov 03 02:31:42.288725 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:42.288612 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:42.329590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:42.329557 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b"] Nov 03 02:31:42.480788 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:42.480744 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:42.629826 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:42.629746 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:42.798734 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:42.798699 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:43.008580 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:31:43.008540 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:31:43.015996 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:43.015976 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Nov 03 02:31:43.016079 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:43.016019 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" podUID="ad2720fd-aa19-4213-a377-c98474134c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Nov 03 02:31:44.016785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:44.016741 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Nov 03 02:31:44.017184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:44.016809 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" podUID="ad2720fd-aa19-4213-a377-c98474134c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Nov 03 02:31:45.016005 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:45.015971 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Nov 03 02:31:45.016211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:45.016025 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" podUID="ad2720fd-aa19-4213-a377-c98474134c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Nov 03 02:31:46.016343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:46.016300 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Nov 03 02:31:46.016751 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:46.016365 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" podUID="ad2720fd-aa19-4213-a377-c98474134c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Nov 03 02:31:47.016054 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.016023 5463 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Nov 03 02:31:47.016232 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.016075 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" podUID="ad2720fd-aa19-4213-a377-c98474134c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Nov 03 02:31:47.227475 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.227422 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw" podUID="06d24e51-c948-4599-b7db-1ea7082bc879" containerName="payload-processing" containerID="cri-o://a261ce48fda2bc6384dadf9f6f99837552287e4c76eaea1948ab59c4381e6dee" gracePeriod=25 Nov 03 02:31:47.330762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.330621 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b" podUID="763ef8e4-77b4-4e99-8bf6-839d76b88c99" containerName="payload-pre-processing" containerID="cri-o://7bb5fbbc0f922f137a5ac4f63923bf2287c25e792c4f11de492644fb55815ee7" gracePeriod=25 Nov 03 02:31:47.381913 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.381837 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:47.717211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.717147 5463 generic.go:356] "Generic (PLEG): container finished" podID="763ef8e4-77b4-4e99-8bf6-839d76b88c99" containerID="7bb5fbbc0f922f137a5ac4f63923bf2287c25e792c4f11de492644fb55815ee7" exitCode=0 Nov 03 02:31:47.718335 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.717291 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b" event={"ID":"763ef8e4-77b4-4e99-8bf6-839d76b88c99","Type":"ContainerDied","Data":"7bb5fbbc0f922f137a5ac4f63923bf2287c25e792c4f11de492644fb55815ee7"} Nov 03 02:31:47.723865 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.721837 5463 generic.go:356] "Generic (PLEG): container finished" podID="06d24e51-c948-4599-b7db-1ea7082bc879" containerID="a261ce48fda2bc6384dadf9f6f99837552287e4c76eaea1948ab59c4381e6dee" exitCode=0 Nov 03 02:31:47.723865 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.722037 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw" event={"ID":"06d24e51-c948-4599-b7db-1ea7082bc879","Type":"ContainerDied","Data":"a261ce48fda2bc6384dadf9f6f99837552287e4c76eaea1948ab59c4381e6dee"} Nov 03 02:31:47.723865 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.722085 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw" event={"ID":"06d24e51-c948-4599-b7db-1ea7082bc879","Type":"ContainerDied","Data":"08bb222450938c6d7c20c682e0175b82b5bab70fa930db564986b7f8e5b109ac"} Nov 03 02:31:47.723865 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.722108 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08bb222450938c6d7c20c682e0175b82b5bab70fa930db564986b7f8e5b109ac" Nov 03 02:31:47.746202 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.737238 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw" Nov 03 02:31:47.776145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.776120 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b" Nov 03 02:31:47.919271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.918535 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d24e51-c948-4599-b7db-1ea7082bc879-kube-api-access-w2q9k\" (UniqueName: \"kubernetes.io/projected/06d24e51-c948-4599-b7db-1ea7082bc879-kube-api-access-w2q9k\") pod \"06d24e51-c948-4599-b7db-1ea7082bc879\" (UID: \"06d24e51-c948-4599-b7db-1ea7082bc879\") " Nov 03 02:31:47.919271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.918701 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/763ef8e4-77b4-4e99-8bf6-839d76b88c99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763ef8e4-77b4-4e99-8bf6-839d76b88c99-plugins-config-volume\") pod \"763ef8e4-77b4-4e99-8bf6-839d76b88c99\" (UID: \"763ef8e4-77b4-4e99-8bf6-839d76b88c99\") " Nov 03 02:31:47.919271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.918766 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06d24e51-c948-4599-b7db-1ea7082bc879-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d24e51-c948-4599-b7db-1ea7082bc879-plugins-config-volume\") pod \"06d24e51-c948-4599-b7db-1ea7082bc879\" (UID: \"06d24e51-c948-4599-b7db-1ea7082bc879\") " Nov 03 02:31:47.919271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.918868 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763ef8e4-77b4-4e99-8bf6-839d76b88c99-kube-api-access-xlvzx\" (UniqueName: \"kubernetes.io/projected/763ef8e4-77b4-4e99-8bf6-839d76b88c99-kube-api-access-xlvzx\") pod \"763ef8e4-77b4-4e99-8bf6-839d76b88c99\" (UID: \"763ef8e4-77b4-4e99-8bf6-839d76b88c99\") " Nov 03 02:31:47.924443 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.921601 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d24e51-c948-4599-b7db-1ea7082bc879-plugins-config-volume" pod "06d24e51-c948-4599-b7db-1ea7082bc879" (UID: "06d24e51-c948-4599-b7db-1ea7082bc879"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:31:47.924443 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.922594 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/763ef8e4-77b4-4e99-8bf6-839d76b88c99-plugins-config-volume" pod "763ef8e4-77b4-4e99-8bf6-839d76b88c99" (UID: "763ef8e4-77b4-4e99-8bf6-839d76b88c99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:31:47.924443 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.924179 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d24e51-c948-4599-b7db-1ea7082bc879-kube-api-access-w2q9k" pod "06d24e51-c948-4599-b7db-1ea7082bc879" (UID: "06d24e51-c948-4599-b7db-1ea7082bc879"). InnerVolumeSpecName "kube-api-access-w2q9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:47.928324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:47.928303 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763ef8e4-77b4-4e99-8bf6-839d76b88c99-kube-api-access-xlvzx" pod "763ef8e4-77b4-4e99-8bf6-839d76b88c99" (UID: "763ef8e4-77b4-4e99-8bf6-839d76b88c99"). InnerVolumeSpecName "kube-api-access-xlvzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:48.021389 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:48.021329 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763ef8e4-77b4-4e99-8bf6-839d76b88c99-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:48.021389 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:48.021377 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d24e51-c948-4599-b7db-1ea7082bc879-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:48.021389 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:48.021396 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlvzx\" (UniqueName: \"kubernetes.io/projected/763ef8e4-77b4-4e99-8bf6-839d76b88c99-kube-api-access-xlvzx\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:48.021766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:48.021412 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2q9k\" (UniqueName: \"kubernetes.io/projected/06d24e51-c948-4599-b7db-1ea7082bc879-kube-api-access-w2q9k\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:48.246364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:48.246335 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:48.246778 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:48.246730 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:48.253225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:48.249449 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:31:48.739636 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:48.739600 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b" event={"ID":"763ef8e4-77b4-4e99-8bf6-839d76b88c99","Type":"ContainerDied","Data":"e7eb05fa13ef0c7be7876c619810d6a8982afce83cb99c2bd50e2d9168cfcac5"} Nov 03 02:31:48.739836 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:48.739681 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b" Nov 03 02:31:48.739836 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:48.739721 5463 scope.go:122] "RemoveContainer" containerID="7bb5fbbc0f922f137a5ac4f63923bf2287c25e792c4f11de492644fb55815ee7" Nov 03 02:31:48.740766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:48.740736 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw" Nov 03 02:31:48.907935 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:48.907900 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b"] Nov 03 02:31:48.911881 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:48.911848 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-6c87669695-ps99b"] Nov 03 02:31:49.021151 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:49.021072 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw"] Nov 03 02:31:49.024124 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:49.024099 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-76c8f7ccf5-wsrrw"] Nov 03 02:31:50.443584 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:50.443550 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06d24e51-c948-4599-b7db-1ea7082bc879" path="/var/lib/kubelet/pods/06d24e51-c948-4599-b7db-1ea7082bc879/volumes" Nov 03 02:31:50.443995 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:50.443980 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="763ef8e4-77b4-4e99-8bf6-839d76b88c99" path="/var/lib/kubelet/pods/763ef8e4-77b4-4e99-8bf6-839d76b88c99/volumes" Nov 03 02:31:51.621113 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:51.621072 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:53.433942 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:53.433907 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:55.566759 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:55.566690 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:55.724250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:55.724175 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:57.143718 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:57.143676 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:57.628522 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:57.628450 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4"] Nov 03 02:31:57.628688 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:57.628662 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" podUID="9df6bd25-71cc-4cd0-bd8e-2e475b759dba" containerName="istio-proxy" containerID="cri-o://f84d4c0317b2616e3640920ee43a050cbf143e61a638534a0d3e770c6b1bd87a" gracePeriod=30 Nov 03 02:31:57.658153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:57.658120 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:58.492640 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:58.492605 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:58.641661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:31:58.641615 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:02.775268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:02.775234 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:32:02.777056 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:02.777032 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:32:02.855616 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:02.855585 5463 generic.go:356] "Generic (PLEG): container finished" podID="9df6bd25-71cc-4cd0-bd8e-2e475b759dba" containerID="f84d4c0317b2616e3640920ee43a050cbf143e61a638534a0d3e770c6b1bd87a" exitCode=0 Nov 03 02:32:02.855805 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:02.855736 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" event={"ID":"9df6bd25-71cc-4cd0-bd8e-2e475b759dba","Type":"ContainerDied","Data":"f84d4c0317b2616e3640920ee43a050cbf143e61a638534a0d3e770c6b1bd87a"} Nov 03 02:32:02.855805 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:02.855779 5463 scope.go:122] "RemoveContainer" containerID="c09469c52ef697cdbb7fb2539340cd21a4d200175d5af78a7ab5bfdf2f5132d2" Nov 03 02:32:02.912387 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:02.912364 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:32:03.002735 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.002704 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-envoy\") pod \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " Nov 03 02:32:03.002924 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.002776 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-workload-certs\") pod \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " Nov 03 02:32:03.002924 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.002833 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-token\" (UniqueName: \"kubernetes.io/projected/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-token\") pod \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " Nov 03 02:32:03.002924 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.002868 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istiod-ca-cert\") pod \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " Nov 03 02:32:03.002924 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.002894 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-workload-socket\") pod \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " Nov 03 02:32:03.003160 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.002939 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-data\") pod \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " Nov 03 02:32:03.003160 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.002980 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-kube-api-access-6qk4g\" (UniqueName: \"kubernetes.io/projected/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-kube-api-access-6qk4g\") pod \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " Nov 03 02:32:03.003160 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.003016 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-credential-socket\") pod \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " Nov 03 02:32:03.003160 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.003062 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-podinfo\") pod \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\" (UID: \"9df6bd25-71cc-4cd0-bd8e-2e475b759dba\") " Nov 03 02:32:03.003160 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.003073 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-workload-certs" pod "9df6bd25-71cc-4cd0-bd8e-2e475b759dba" (UID: "9df6bd25-71cc-4cd0-bd8e-2e475b759dba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:32:03.003464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.003377 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-data" pod "9df6bd25-71cc-4cd0-bd8e-2e475b759dba" (UID: "9df6bd25-71cc-4cd0-bd8e-2e475b759dba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:32:03.003464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.003395 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istiod-ca-cert" pod "9df6bd25-71cc-4cd0-bd8e-2e475b759dba" (UID: "9df6bd25-71cc-4cd0-bd8e-2e475b759dba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:32:03.003464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.003395 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-credential-socket" pod "9df6bd25-71cc-4cd0-bd8e-2e475b759dba" (UID: "9df6bd25-71cc-4cd0-bd8e-2e475b759dba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:32:03.003703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.003569 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:03.003703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.003591 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:03.003703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.003605 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:03.003703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.003618 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:03.003703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.003674 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-workload-socket" pod "9df6bd25-71cc-4cd0-bd8e-2e475b759dba" (UID: "9df6bd25-71cc-4cd0-bd8e-2e475b759dba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:32:03.005483 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.005462 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-podinfo" pod "9df6bd25-71cc-4cd0-bd8e-2e475b759dba" (UID: "9df6bd25-71cc-4cd0-bd8e-2e475b759dba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:32:03.005590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.005478 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-kube-api-access-6qk4g" pod "9df6bd25-71cc-4cd0-bd8e-2e475b759dba" (UID: "9df6bd25-71cc-4cd0-bd8e-2e475b759dba"). InnerVolumeSpecName "kube-api-access-6qk4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:32:03.005590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.005522 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-envoy" pod "9df6bd25-71cc-4cd0-bd8e-2e475b759dba" (UID: "9df6bd25-71cc-4cd0-bd8e-2e475b759dba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:32:03.005669 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.005632 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-token" pod "9df6bd25-71cc-4cd0-bd8e-2e475b759dba" (UID: "9df6bd25-71cc-4cd0-bd8e-2e475b759dba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:32:03.104589 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.104560 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:03.104589 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.104590 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:03.104804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.104604 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qk4g\" (UniqueName: \"kubernetes.io/projected/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-kube-api-access-6qk4g\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:03.104804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.104615 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:03.104804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.104625 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df6bd25-71cc-4cd0-bd8e-2e475b759dba-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:03.438501 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.438467 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:03.866686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.866589 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" Nov 03 02:32:03.866686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.866595 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4" event={"ID":"9df6bd25-71cc-4cd0-bd8e-2e475b759dba","Type":"ContainerDied","Data":"5dad18d8acdef5be5bd37e9ebc356f50cadffc0db23c8f9f0215178af450f427"} Nov 03 02:32:03.867251 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:03.866730 5463 scope.go:122] "RemoveContainer" containerID="f84d4c0317b2616e3640920ee43a050cbf143e61a638534a0d3e770c6b1bd87a" Nov 03 02:32:04.083255 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:04.083222 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4"] Nov 03 02:32:04.133005 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:04.132938 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-7f760c-openshift-default-5fdfc6f8fc-mlsd4"] Nov 03 02:32:04.143544 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:04.143515 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:04.442841 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:04.442805 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df6bd25-71cc-4cd0-bd8e-2e475b759dba" path="/var/lib/kubelet/pods/9df6bd25-71cc-4cd0-bd8e-2e475b759dba/volumes" Nov 03 02:32:04.532123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:04.532089 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv"] Nov 03 02:32:04.909620 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:04.909573 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:05.064924 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:05.064842 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:06.188647 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.188615 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz"] Nov 03 02:32:06.189177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.189161 5463 cpu_manager.go:414] "containerMap: removing container" podUID="763ef8e4-77b4-4e99-8bf6-839d76b88c99" containerName="payload-pre-processing" Nov 03 02:32:06.189271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.189177 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763ef8e4-77b4-4e99-8bf6-839d76b88c99" containerName="payload-pre-processing" Nov 03 02:32:06.189271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.189206 5463 cpu_manager.go:414] "containerMap: removing container" podUID="9df6bd25-71cc-4cd0-bd8e-2e475b759dba" containerName="istio-proxy" Nov 03 02:32:06.189271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.189212 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df6bd25-71cc-4cd0-bd8e-2e475b759dba" containerName="istio-proxy" Nov 03 02:32:06.189271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.189245 5463 cpu_manager.go:414] "containerMap: removing container" podUID="06d24e51-c948-4599-b7db-1ea7082bc879" containerName="payload-processing" Nov 03 02:32:06.189271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.189253 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d24e51-c948-4599-b7db-1ea7082bc879" containerName="payload-processing" Nov 03 02:32:06.189551 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.189369 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df6bd25-71cc-4cd0-bd8e-2e475b759dba" containerName="istio-proxy" Nov 03 02:32:06.189551 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.189380 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="763ef8e4-77b4-4e99-8bf6-839d76b88c99" containerName="payload-pre-processing" Nov 03 02:32:06.189551 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.189395 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d24e51-c948-4599-b7db-1ea7082bc879" containerName="payload-processing" Nov 03 02:32:06.193472 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.193448 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.240652 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.239453 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr"] Nov 03 02:32:06.240652 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.240129 5463 cpu_manager.go:414] "containerMap: removing container" podUID="9df6bd25-71cc-4cd0-bd8e-2e475b759dba" containerName="istio-proxy" Nov 03 02:32:06.240652 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.240150 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df6bd25-71cc-4cd0-bd8e-2e475b759dba" containerName="istio-proxy" Nov 03 02:32:06.240652 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.240359 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df6bd25-71cc-4cd0-bd8e-2e475b759dba" containerName="istio-proxy" Nov 03 02:32:06.242662 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.242630 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.242814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.242715 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.242814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.242802 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.242934 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.242850 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.242934 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.242876 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxmvq\" (UniqueName: \"kubernetes.io/projected/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-kube-api-access-kxmvq\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.242934 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.242904 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.243107 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.242972 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.243107 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.242996 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.243233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.243112 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.243233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.243141 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.286814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.286784 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz"] Nov 03 02:32:06.333601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.333567 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr"] Nov 03 02:32:06.344598 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.344567 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.344760 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.344697 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.344830 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.344790 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.344892 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.344837 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.344948 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.344888 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.345080 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.345055 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.345152 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.345098 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxmvq\" (UniqueName: \"kubernetes.io/projected/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-kube-api-access-kxmvq\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.345152 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.345128 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.345152 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.345131 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.345337 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.345209 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2m6w\" (UniqueName: \"kubernetes.io/projected/87ab8925-cc59-4bf1-a804-051b4f7907b3-kube-api-access-m2m6w\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.345337 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.345313 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.345460 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.345345 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ab8925-cc59-4bf1-a804-051b4f7907b3-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.345460 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.345381 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.345460 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.345431 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.345460 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.345450 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.345661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.345491 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.345661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.345552 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.345661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.345578 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.345661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.345611 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.345661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.345653 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.345987 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.345882 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.346123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.346097 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.346832 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.346808 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.347436 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.347389 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.348424 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.348399 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.401847 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.401816 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.402000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.401981 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxmvq\" (UniqueName: \"kubernetes.io/projected/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-kube-api-access-kxmvq\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.446838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.446773 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.446838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.446829 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2m6w\" (UniqueName: \"kubernetes.io/projected/87ab8925-cc59-4bf1-a804-051b4f7907b3-kube-api-access-m2m6w\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.447042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.446864 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.447042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.446881 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ab8925-cc59-4bf1-a804-051b4f7907b3-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.447042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.446900 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.447042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.446930 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.447042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.446979 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.447042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.447004 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.447421 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.447168 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.447421 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.447252 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.447421 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.447291 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.447606 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.447563 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ab8925-cc59-4bf1-a804-051b4f7907b3-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.447666 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.447629 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.447666 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.447654 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.449051 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.449023 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.449447 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.449429 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.503497 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.503476 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.504376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.504355 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2m6w\" (UniqueName: \"kubernetes.io/projected/87ab8925-cc59-4bf1-a804-051b4f7907b3-kube-api-access-m2m6w\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.514237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.514184 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:06.559336 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.559307 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:06.932632 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:32:06.932581 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce481b0b_ac96_4898_bcfd_e1afa0a3bcb8.slice/crio-ded5eeebde6f682b676cce49ae4562ec25dbb78b923d2c5e4236297682aabe39 WatchSource:0}: Error finding container ded5eeebde6f682b676cce49ae4562ec25dbb78b923d2c5e4236297682aabe39: Status 404 returned error can't find the container with id ded5eeebde6f682b676cce49ae4562ec25dbb78b923d2c5e4236297682aabe39 Nov 03 02:32:06.935057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.935017 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:32:06.935174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.935082 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:32:06.935174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.935112 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:32:06.936393 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.936371 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz"] Nov 03 02:32:06.985943 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.985611 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr"] Nov 03 02:32:06.991103 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:32:06.991063 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ab8925_cc59_4bf1_a804_051b4f7907b3.slice/crio-06a118f4a78e2dc0415fa45ea7c2721cf441877f2822cb8c11cc873407d131e6 WatchSource:0}: Error finding container 06a118f4a78e2dc0415fa45ea7c2721cf441877f2822cb8c11cc873407d131e6: Status 404 returned error can't find the container with id 06a118f4a78e2dc0415fa45ea7c2721cf441877f2822cb8c11cc873407d131e6 Nov 03 02:32:06.993726 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.993688 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:32:06.993937 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.993768 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:32:06.993937 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:06.993810 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:32:07.903394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:07.903342 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" event={"ID":"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8","Type":"ContainerStarted","Data":"3f99e93d1f9a326ed9aab31255056864ceaaf63f136e7cebe0a6cd99b2046e0f"} Nov 03 02:32:07.904066 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:07.904037 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" event={"ID":"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8","Type":"ContainerStarted","Data":"ded5eeebde6f682b676cce49ae4562ec25dbb78b923d2c5e4236297682aabe39"} Nov 03 02:32:07.905913 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:07.905885 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" event={"ID":"87ab8925-cc59-4bf1-a804-051b4f7907b3","Type":"ContainerStarted","Data":"58c282afb041c49feffa8b9b50400873fc9c6eb1555a86997d4c7357ae50c88c"} Nov 03 02:32:07.906065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:07.905938 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" event={"ID":"87ab8925-cc59-4bf1-a804-051b4f7907b3","Type":"ContainerStarted","Data":"06a118f4a78e2dc0415fa45ea7c2721cf441877f2822cb8c11cc873407d131e6"} Nov 03 02:32:08.081373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:08.081298 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" podStartSLOduration=2.081278261 podStartE2EDuration="2.081278261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:32:08.078702429 +0000 UTC m=+2409.762811585" watchObservedRunningTime="2026-11-03 02:32:08.081278261 +0000 UTC m=+2409.765387417" Nov 03 02:32:08.217980 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:08.215775 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" podStartSLOduration=2.215754156 podStartE2EDuration="2.215754156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:32:08.211686144 +0000 UTC m=+2409.895795299" watchObservedRunningTime="2026-11-03 02:32:08.215754156 +0000 UTC m=+2409.899863312" Nov 03 02:32:08.515853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:08.515756 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:08.517980 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:08.517941 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Nov 03 02:32:08.518253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:08.518222 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" podUID="ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Nov 03 02:32:08.529127 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:08.529102 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:08.562688 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:08.562646 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:08.566705 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:08.566672 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 03 02:32:08.566954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:08.566927 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" podUID="87ab8925-cc59-4bf1-a804-051b4f7907b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 03 02:32:09.515332 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:09.515298 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Nov 03 02:32:09.515775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:09.515349 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" podUID="ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Nov 03 02:32:09.532879 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:09.532839 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv" podUID="6421252b-76d2-40ed-af4c-9e8086e8b87b" containerName="payload-processing" containerID="cri-o://0f81047507d04a713129c101a7eac62aa71337fc9959de90b76020d3aea4342f" gracePeriod=25 Nov 03 02:32:09.568223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:09.565536 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 03 02:32:09.568223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:09.565594 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" podUID="87ab8925-cc59-4bf1-a804-051b4f7907b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 03 02:32:09.783272 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:32:09.782782 5463 logging.go:55] [core] [Channel #7065 SubChannel #7066]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.203:50052", ServerName: "10.133.0.203:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.203:50052: connect: connection refused" Nov 03 02:32:09.939237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:09.934736 5463 generic.go:356] "Generic (PLEG): container finished" podID="6421252b-76d2-40ed-af4c-9e8086e8b87b" containerID="0f81047507d04a713129c101a7eac62aa71337fc9959de90b76020d3aea4342f" exitCode=0 Nov 03 02:32:09.939237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:09.934899 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv" event={"ID":"6421252b-76d2-40ed-af4c-9e8086e8b87b","Type":"ContainerDied","Data":"0f81047507d04a713129c101a7eac62aa71337fc9959de90b76020d3aea4342f"} Nov 03 02:32:10.033265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:10.033237 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv" Nov 03 02:32:10.046924 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:10.046844 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6421252b-76d2-40ed-af4c-9e8086e8b87b-kube-api-access-wtrck\" (UniqueName: \"kubernetes.io/projected/6421252b-76d2-40ed-af4c-9e8086e8b87b-kube-api-access-wtrck\") pod \"6421252b-76d2-40ed-af4c-9e8086e8b87b\" (UID: \"6421252b-76d2-40ed-af4c-9e8086e8b87b\") " Nov 03 02:32:10.047072 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:10.046921 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6421252b-76d2-40ed-af4c-9e8086e8b87b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6421252b-76d2-40ed-af4c-9e8086e8b87b-plugins-config-volume\") pod \"6421252b-76d2-40ed-af4c-9e8086e8b87b\" (UID: \"6421252b-76d2-40ed-af4c-9e8086e8b87b\") " Nov 03 02:32:10.091228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:10.055466 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6421252b-76d2-40ed-af4c-9e8086e8b87b-plugins-config-volume" pod "6421252b-76d2-40ed-af4c-9e8086e8b87b" (UID: "6421252b-76d2-40ed-af4c-9e8086e8b87b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:32:10.091228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:10.076740 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6421252b-76d2-40ed-af4c-9e8086e8b87b-kube-api-access-wtrck" pod "6421252b-76d2-40ed-af4c-9e8086e8b87b" (UID: "6421252b-76d2-40ed-af4c-9e8086e8b87b"). InnerVolumeSpecName "kube-api-access-wtrck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:32:10.157217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:10.152882 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtrck\" (UniqueName: \"kubernetes.io/projected/6421252b-76d2-40ed-af4c-9e8086e8b87b-kube-api-access-wtrck\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:10.157217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:10.152919 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6421252b-76d2-40ed-af4c-9e8086e8b87b-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:10.321271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:10.318011 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:10.515371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:10.515338 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Nov 03 02:32:10.515796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:10.515390 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" podUID="ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Nov 03 02:32:10.567218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:10.562557 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 03 02:32:10.567218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:10.562623 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" podUID="87ab8925-cc59-4bf1-a804-051b4f7907b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 03 02:32:10.783216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:10.782675 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv" podUID="6421252b-76d2-40ed-af4c-9e8086e8b87b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.203:50052\" within 1s: context deadline exceeded" Nov 03 02:32:10.944459 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:10.943726 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv" Nov 03 02:32:10.944459 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:10.943772 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv" event={"ID":"6421252b-76d2-40ed-af4c-9e8086e8b87b","Type":"ContainerDied","Data":"ba771429317dc5b2b82a345a17c1778faa8a746e9533ea1eb3c615270cbd5c47"} Nov 03 02:32:10.944459 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:10.943845 5463 scope.go:122] "RemoveContainer" containerID="0f81047507d04a713129c101a7eac62aa71337fc9959de90b76020d3aea4342f" Nov 03 02:32:11.138272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.131640 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv"] Nov 03 02:32:11.183214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.183120 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-711318-675db99787-mk5zv"] Nov 03 02:32:11.514728 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.514695 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Nov 03 02:32:11.514961 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.514745 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" podUID="ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Nov 03 02:32:11.564216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.563630 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 03 02:32:11.564216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.563706 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" podUID="87ab8925-cc59-4bf1-a804-051b4f7907b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 03 02:32:11.568213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.566557 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn"] Nov 03 02:32:11.568213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.567444 5463 cpu_manager.go:414] "containerMap: removing container" podUID="6421252b-76d2-40ed-af4c-9e8086e8b87b" containerName="payload-processing" Nov 03 02:32:11.568213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.567464 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6421252b-76d2-40ed-af4c-9e8086e8b87b" containerName="payload-processing" Nov 03 02:32:11.568213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.567651 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="6421252b-76d2-40ed-af4c-9e8086e8b87b" containerName="payload-processing" Nov 03 02:32:11.578211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.575250 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.622655 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.622627 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn"] Nov 03 02:32:11.631324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.630776 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0ba391-497c-4575-909a-10a65432678d-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.631324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.630812 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m9zl\" (UniqueName: \"kubernetes.io/projected/4d0ba391-497c-4575-909a-10a65432678d-kube-api-access-7m9zl\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.631324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.630833 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d0ba391-497c-4575-909a-10a65432678d-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.631324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.630880 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.631324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.630936 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.631324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.630967 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0ba391-497c-4575-909a-10a65432678d-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.631324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.631114 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.631324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.631140 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.631324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.631228 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.668503 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.668470 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-1bccbd-openshift-default-dockercfg-c57w7\"" Nov 03 02:32:11.732266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.732230 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.732573 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.732549 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.732721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.732703 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0ba391-497c-4575-909a-10a65432678d-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.732929 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.732912 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.733465 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.733451 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.733684 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.733463 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.733684 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.733412 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.733840 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.733615 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.733840 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.733778 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0ba391-497c-4575-909a-10a65432678d-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.733946 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.733875 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0ba391-497c-4575-909a-10a65432678d-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.733946 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.733901 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m9zl\" (UniqueName: \"kubernetes.io/projected/4d0ba391-497c-4575-909a-10a65432678d-kube-api-access-7m9zl\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.733946 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.733912 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.733946 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.733924 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d0ba391-497c-4575-909a-10a65432678d-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.734357 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.734329 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.736428 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.736404 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.740226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.739789 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0ba391-497c-4575-909a-10a65432678d-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.819058 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.818980 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d0ba391-497c-4575-909a-10a65432678d-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.821454 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.821409 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m9zl\" (UniqueName: \"kubernetes.io/projected/4d0ba391-497c-4575-909a-10a65432678d-kube-api-access-7m9zl\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:11.913369 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:11.913015 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:12.411598 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:12.411565 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn"] Nov 03 02:32:12.479227 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:32:12.469980 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0ba391_497c_4575_909a_10a65432678d.slice/crio-a586a3387e3ea79d6a20fd52380d6308ad52e21c5e5addfe53c77dff691ce3a1 WatchSource:0}: Error finding container a586a3387e3ea79d6a20fd52380d6308ad52e21c5e5addfe53c77dff691ce3a1: Status 404 returned error can't find the container with id a586a3387e3ea79d6a20fd52380d6308ad52e21c5e5addfe53c77dff691ce3a1 Nov 03 02:32:12.509448 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:12.506072 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:32:12.509448 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:12.506178 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:32:12.509448 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:12.508677 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:32:12.511651 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:12.511617 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6421252b-76d2-40ed-af4c-9e8086e8b87b" path="/var/lib/kubelet/pods/6421252b-76d2-40ed-af4c-9e8086e8b87b/volumes" Nov 03 02:32:12.514808 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:12.514775 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Nov 03 02:32:12.514925 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:12.514854 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" podUID="ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Nov 03 02:32:12.569218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:12.569150 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 03 02:32:12.570435 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:12.570235 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" podUID="87ab8925-cc59-4bf1-a804-051b4f7907b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 03 02:32:13.022224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:13.021443 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" event={"ID":"4d0ba391-497c-4575-909a-10a65432678d","Type":"ContainerStarted","Data":"de5df43669c4345c1b55cadf63cf6a5283762a5c14a1ab83aee295a947c2ed06"} Nov 03 02:32:13.022224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:13.021511 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" event={"ID":"4d0ba391-497c-4575-909a-10a65432678d","Type":"ContainerStarted","Data":"a586a3387e3ea79d6a20fd52380d6308ad52e21c5e5addfe53c77dff691ce3a1"} Nov 03 02:32:13.045764 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:32:13.045728 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:32:13.145456 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:13.143688 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" podStartSLOduration=2.143673423 podStartE2EDuration="2.143673423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:32:13.141623993 +0000 UTC m=+2414.825733148" watchObservedRunningTime="2026-11-03 02:32:13.143673423 +0000 UTC m=+2414.827782577" Nov 03 02:32:13.515564 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:13.515520 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Nov 03 02:32:13.515818 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:13.515790 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" podUID="ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Nov 03 02:32:13.562226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:13.560643 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 03 02:32:13.562226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:13.560705 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" podUID="87ab8925-cc59-4bf1-a804-051b4f7907b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 03 02:32:13.633694 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:13.633662 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:13.916549 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:13.916496 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:13.926529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:13.926480 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 03 02:32:13.926852 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:13.926802 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" podUID="4d0ba391-497c-4575-909a-10a65432678d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 03 02:32:14.515103 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:14.515067 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Nov 03 02:32:14.515349 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:14.515134 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" podUID="ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Nov 03 02:32:14.565862 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:14.565826 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 03 02:32:14.566145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:14.566115 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" podUID="87ab8925-cc59-4bf1-a804-051b4f7907b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 03 02:32:14.916222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:14.914635 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 03 02:32:14.916222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:14.914696 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" podUID="4d0ba391-497c-4575-909a-10a65432678d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 03 02:32:15.923221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:15.917140 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 03 02:32:15.923221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:15.917234 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" podUID="4d0ba391-497c-4575-909a-10a65432678d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 03 02:32:16.516997 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:16.516944 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:32:16.517256 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:16.517042 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" podUID="ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:32:16.571216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:16.570286 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:32:16.571216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:16.570357 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" podUID="87ab8925-cc59-4bf1-a804-051b4f7907b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:32:16.597052 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:16.596990 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:16.932216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:16.931919 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 03 02:32:16.932216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:16.931990 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" podUID="4d0ba391-497c-4575-909a-10a65432678d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 03 02:32:17.521358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:17.521303 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:32:17.521828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:17.521653 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" podUID="ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:32:17.537155 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:17.537127 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:17.538366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:17.538238 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:17.616138 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:17.616099 5463 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:32:17.616582 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:17.616469 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:32:17.616785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:17.616543 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" podUID="87ab8925-cc59-4bf1-a804-051b4f7907b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:32:17.642613 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:17.642584 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:17.644162 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:17.644140 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:17.720790 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:17.720754 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:32:17.914061 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:17.914030 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 03 02:32:17.914220 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:17.914079 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" podUID="4d0ba391-497c-4575-909a-10a65432678d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 03 02:32:18.914321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:18.914288 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 03 02:32:18.914786 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:18.914355 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" podUID="4d0ba391-497c-4575-909a-10a65432678d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 03 02:32:19.024836 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:19.024802 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:19.913980 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:19.913938 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 03 02:32:19.914147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:19.914008 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" podUID="4d0ba391-497c-4575-909a-10a65432678d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 03 02:32:20.559235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:20.557879 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:21.836042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:21.836010 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:21.836556 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:21.836491 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:21.837544 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:21.837522 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:32:24.121477 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:24.121441 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:24.612763 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:24.612727 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:25.707315 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:25.707280 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k"] Nov 03 02:32:25.710751 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:25.710730 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k" Nov 03 02:32:25.763647 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:25.763606 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f9bff9\"" Nov 03 02:32:25.764903 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:25.764876 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k"] Nov 03 02:32:25.791313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:25.791281 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn9tp\" (UniqueName: \"kubernetes.io/projected/411eec37-7cf8-4243-b521-93c3da337d3d-kube-api-access-kn9tp\") pod \"payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k\" (UID: \"411eec37-7cf8-4243-b521-93c3da337d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k" Nov 03 02:32:25.792278 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:25.792252 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/411eec37-7cf8-4243-b521-93c3da337d3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k\" (UID: \"411eec37-7cf8-4243-b521-93c3da337d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k" Nov 03 02:32:25.812247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:25.812223 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f9bff9-dockercfg-tn74j\"" Nov 03 02:32:25.893712 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:25.893676 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn9tp\" (UniqueName: \"kubernetes.io/projected/411eec37-7cf8-4243-b521-93c3da337d3d-kube-api-access-kn9tp\") pod \"payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k\" (UID: \"411eec37-7cf8-4243-b521-93c3da337d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k" Nov 03 02:32:25.893933 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:25.893899 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/411eec37-7cf8-4243-b521-93c3da337d3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k\" (UID: \"411eec37-7cf8-4243-b521-93c3da337d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k" Nov 03 02:32:25.894673 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:25.894650 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/411eec37-7cf8-4243-b521-93c3da337d3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k\" (UID: \"411eec37-7cf8-4243-b521-93c3da337d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k" Nov 03 02:32:25.954071 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:25.954036 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn9tp\" (UniqueName: \"kubernetes.io/projected/411eec37-7cf8-4243-b521-93c3da337d3d-kube-api-access-kn9tp\") pod \"payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k\" (UID: \"411eec37-7cf8-4243-b521-93c3da337d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k" Nov 03 02:32:26.025767 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:26.025686 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k" Nov 03 02:32:26.401356 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:32:26.401322 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411eec37_7cf8_4243_b521_93c3da337d3d.slice/crio-b4c9449ea2d721d2619b9ab1f5574e00c08be4a4e21db0cacf9c2384aa3b2503 WatchSource:0}: Error finding container b4c9449ea2d721d2619b9ab1f5574e00c08be4a4e21db0cacf9c2384aa3b2503: Status 404 returned error can't find the container with id b4c9449ea2d721d2619b9ab1f5574e00c08be4a4e21db0cacf9c2384aa3b2503 Nov 03 02:32:26.406025 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:26.405921 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k"] Nov 03 02:32:26.483596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:26.483569 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj"] Nov 03 02:32:26.487653 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:26.487633 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj" Nov 03 02:32:26.545318 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:26.545280 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj"] Nov 03 02:32:26.604556 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:26.604519 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjsqq\" (UniqueName: \"kubernetes.io/projected/4c0427d1-e64c-4d1f-9169-9637e9756507-kube-api-access-xjsqq\") pod \"payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj\" (UID: \"4c0427d1-e64c-4d1f-9169-9637e9756507\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj" Nov 03 02:32:26.604838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:26.604815 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c0427d1-e64c-4d1f-9169-9637e9756507-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj\" (UID: \"4c0427d1-e64c-4d1f-9169-9637e9756507\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj" Nov 03 02:32:26.706373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:26.706280 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjsqq\" (UniqueName: \"kubernetes.io/projected/4c0427d1-e64c-4d1f-9169-9637e9756507-kube-api-access-xjsqq\") pod \"payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj\" (UID: \"4c0427d1-e64c-4d1f-9169-9637e9756507\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj" Nov 03 02:32:26.706532 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:26.706472 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c0427d1-e64c-4d1f-9169-9637e9756507-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj\" (UID: \"4c0427d1-e64c-4d1f-9169-9637e9756507\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj" Nov 03 02:32:26.707299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:26.707274 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c0427d1-e64c-4d1f-9169-9637e9756507-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj\" (UID: \"4c0427d1-e64c-4d1f-9169-9637e9756507\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj" Nov 03 02:32:26.765038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:26.765005 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjsqq\" (UniqueName: \"kubernetes.io/projected/4c0427d1-e64c-4d1f-9169-9637e9756507-kube-api-access-xjsqq\") pod \"payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj\" (UID: \"4c0427d1-e64c-4d1f-9169-9637e9756507\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj" Nov 03 02:32:26.803128 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:26.803103 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj" Nov 03 02:32:27.177463 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:32:27.177422 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c0427d1_e64c_4d1f_9169_9637e9756507.slice/crio-9d98e2dde734150872b70694297f9ac6e7dcb5aa2b03dd6e800486864f58cbee WatchSource:0}: Error finding container 9d98e2dde734150872b70694297f9ac6e7dcb5aa2b03dd6e800486864f58cbee: Status 404 returned error can't find the container with id 9d98e2dde734150872b70694297f9ac6e7dcb5aa2b03dd6e800486864f58cbee Nov 03 02:32:27.179060 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:27.179024 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj"] Nov 03 02:32:27.241954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:27.241928 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:27.360618 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:27.360572 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj" event={"ID":"4c0427d1-e64c-4d1f-9169-9637e9756507","Type":"ContainerStarted","Data":"7295fc25c9253d82936153419cdec49558296800589c829eaac8adebffe9b472"} Nov 03 02:32:27.360815 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:27.360632 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj" event={"ID":"4c0427d1-e64c-4d1f-9169-9637e9756507","Type":"ContainerStarted","Data":"9d98e2dde734150872b70694297f9ac6e7dcb5aa2b03dd6e800486864f58cbee"} Nov 03 02:32:27.360815 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:27.360656 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj" Nov 03 02:32:27.362274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:27.362247 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k" event={"ID":"411eec37-7cf8-4243-b521-93c3da337d3d","Type":"ContainerStarted","Data":"c98513b11ffe1ab120e40997c521f4d317f6ada1f25e6ef7e34fc6f89deec00f"} Nov 03 02:32:27.362389 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:27.362288 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k" event={"ID":"411eec37-7cf8-4243-b521-93c3da337d3d","Type":"ContainerStarted","Data":"b4c9449ea2d721d2619b9ab1f5574e00c08be4a4e21db0cacf9c2384aa3b2503"} Nov 03 02:32:27.362389 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:27.362324 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k" Nov 03 02:32:27.479007 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:27.478914 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj" podStartSLOduration=1.478899847 podStartE2EDuration="1.478899847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:32:27.477255844 +0000 UTC m=+2429.161365001" watchObservedRunningTime="2026-11-03 02:32:27.478899847 +0000 UTC m=+2429.163009001" Nov 03 02:32:27.598219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:27.598133 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k" podStartSLOduration=2.598113874 podStartE2EDuration="2.598113874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:32:27.593894485 +0000 UTC m=+2429.278003640" watchObservedRunningTime="2026-11-03 02:32:27.598113874 +0000 UTC m=+2429.282223039" Nov 03 02:32:32.024998 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:32.024960 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:36.736449 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:36.736403 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:38.371755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:38.371716 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj" Nov 03 02:32:38.372225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:38.371795 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k" Nov 03 02:32:41.826271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:41.825167 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:43.008224 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:32:43.008171 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:32:45.009727 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:45.009698 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj_4c0427d1-e64c-4d1f-9169-9637e9756507/payload-processing/0.log" Nov 03 02:32:45.930748 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:45.930711 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:46.218141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:46.218040 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:47.740590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:47.740555 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:52.685141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:52.685083 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:53.067918 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:53.067824 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:53.310255 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:53.310212 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:54.695529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:54.695493 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:55.227670 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:55.227638 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:57.309376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.309338 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m"] Nov 03 02:32:57.309837 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.309592 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" podUID="6d372c46-0419-4d51-a4e6-0e651b656331" containerName="maas-api" containerID="cri-o://72de326e507553fb8b26d057c959a1d98bf29ae39ccbc9cd6b4b8157eded1d07" gracePeriod=30 Nov 03 02:32:57.459001 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.458969 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:57.578890 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.578827 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" Nov 03 02:32:57.593112 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.593077 5463 generic.go:356] "Generic (PLEG): container finished" podID="6d372c46-0419-4d51-a4e6-0e651b656331" containerID="72de326e507553fb8b26d057c959a1d98bf29ae39ccbc9cd6b4b8157eded1d07" exitCode=0 Nov 03 02:32:57.593355 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.593130 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" Nov 03 02:32:57.593501 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.593147 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" event={"ID":"6d372c46-0419-4d51-a4e6-0e651b656331","Type":"ContainerDied","Data":"72de326e507553fb8b26d057c959a1d98bf29ae39ccbc9cd6b4b8157eded1d07"} Nov 03 02:32:57.593681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.593657 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m" event={"ID":"6d372c46-0419-4d51-a4e6-0e651b656331","Type":"ContainerDied","Data":"b077191daaea2f864026f672e33ee2330296f1123ba196ae16b656f3b85ac600"} Nov 03 02:32:57.593842 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.593827 5463 scope.go:122] "RemoveContainer" containerID="72de326e507553fb8b26d057c959a1d98bf29ae39ccbc9cd6b4b8157eded1d07" Nov 03 02:32:57.609706 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.609604 5463 scope.go:122] "RemoveContainer" containerID="72de326e507553fb8b26d057c959a1d98bf29ae39ccbc9cd6b4b8157eded1d07" Nov 03 02:32:57.610130 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:32:57.610106 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72de326e507553fb8b26d057c959a1d98bf29ae39ccbc9cd6b4b8157eded1d07\": container with ID starting with 72de326e507553fb8b26d057c959a1d98bf29ae39ccbc9cd6b4b8157eded1d07 not found: ID does not exist" containerID="72de326e507553fb8b26d057c959a1d98bf29ae39ccbc9cd6b4b8157eded1d07" Nov 03 02:32:57.610304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.610143 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72de326e507553fb8b26d057c959a1d98bf29ae39ccbc9cd6b4b8157eded1d07"} err="failed to get container status \"72de326e507553fb8b26d057c959a1d98bf29ae39ccbc9cd6b4b8157eded1d07\": rpc error: code = NotFound desc = could not find container \"72de326e507553fb8b26d057c959a1d98bf29ae39ccbc9cd6b4b8157eded1d07\": container with ID starting with 72de326e507553fb8b26d057c959a1d98bf29ae39ccbc9cd6b4b8157eded1d07 not found: ID does not exist" Nov 03 02:32:57.616916 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.616880 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d372c46-0419-4d51-a4e6-0e651b656331-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d372c46-0419-4d51-a4e6-0e651b656331-metrics-certs\") pod \"6d372c46-0419-4d51-a4e6-0e651b656331\" (UID: \"6d372c46-0419-4d51-a4e6-0e651b656331\") " Nov 03 02:32:57.617096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.617076 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d372c46-0419-4d51-a4e6-0e651b656331-kube-api-access-nbzbq\" (UniqueName: \"kubernetes.io/projected/6d372c46-0419-4d51-a4e6-0e651b656331-kube-api-access-nbzbq\") pod \"6d372c46-0419-4d51-a4e6-0e651b656331\" (UID: \"6d372c46-0419-4d51-a4e6-0e651b656331\") " Nov 03 02:32:57.617262 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.617129 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d372c46-0419-4d51-a4e6-0e651b656331-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d372c46-0419-4d51-a4e6-0e651b656331-openshift-service-ca\") pod \"6d372c46-0419-4d51-a4e6-0e651b656331\" (UID: \"6d372c46-0419-4d51-a4e6-0e651b656331\") " Nov 03 02:32:57.617262 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.617178 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d372c46-0419-4d51-a4e6-0e651b656331-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d372c46-0419-4d51-a4e6-0e651b656331-maas-api-tls\") pod \"6d372c46-0419-4d51-a4e6-0e651b656331\" (UID: \"6d372c46-0419-4d51-a4e6-0e651b656331\") " Nov 03 02:32:57.620027 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.620000 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d372c46-0419-4d51-a4e6-0e651b656331-openshift-service-ca" pod "6d372c46-0419-4d51-a4e6-0e651b656331" (UID: "6d372c46-0419-4d51-a4e6-0e651b656331"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:32:57.622661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.622637 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d372c46-0419-4d51-a4e6-0e651b656331-maas-api-tls" pod "6d372c46-0419-4d51-a4e6-0e651b656331" (UID: "6d372c46-0419-4d51-a4e6-0e651b656331"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:32:57.622764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.622640 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d372c46-0419-4d51-a4e6-0e651b656331-kube-api-access-nbzbq" pod "6d372c46-0419-4d51-a4e6-0e651b656331" (UID: "6d372c46-0419-4d51-a4e6-0e651b656331"). InnerVolumeSpecName "kube-api-access-nbzbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:32:57.624303 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.624263 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d372c46-0419-4d51-a4e6-0e651b656331-metrics-certs" pod "6d372c46-0419-4d51-a4e6-0e651b656331" (UID: "6d372c46-0419-4d51-a4e6-0e651b656331"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:32:57.719843 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.719805 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbzbq\" (UniqueName: \"kubernetes.io/projected/6d372c46-0419-4d51-a4e6-0e651b656331-kube-api-access-nbzbq\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:57.720001 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.719848 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d372c46-0419-4d51-a4e6-0e651b656331-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:57.720001 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.719864 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d372c46-0419-4d51-a4e6-0e651b656331-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:57.720001 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.719876 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d372c46-0419-4d51-a4e6-0e651b656331-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:57.871355 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.871267 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52"] Nov 03 02:32:57.871543 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.871464 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52" podUID="d9153ab8-8061-4275-b9ad-8aea6e949ab7" containerName="payload-processing" containerID="cri-o://36be79d2dd5147f499b052916c1061cb364d403a09f1bfcd46f0bb53c3283b02" gracePeriod=30 Nov 03 02:32:57.921281 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.921252 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v"] Nov 03 02:32:57.921535 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:57.921482 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v" podUID="14e3d223-26f5-43d9-9f48-1818d0bfe12e" containerName="payload-pre-processing" containerID="cri-o://b08a9f479ac5100bb210533950056ac7412a36eb89cfdf4ffd0d695072d34b71" gracePeriod=30 Nov 03 02:32:58.051556 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.051514 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:58.119236 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.113450 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m"] Nov 03 02:32:58.122893 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.122635 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb1940-56cdf8c887-2cq8m"] Nov 03 02:32:58.198995 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.198966 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52" Nov 03 02:32:58.211578 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.211549 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:58.227246 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.227218 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v" Nov 03 02:32:58.326607 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.326572 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14e3d223-26f5-43d9-9f48-1818d0bfe12e-kube-api-access-x42nq\" (UniqueName: \"kubernetes.io/projected/14e3d223-26f5-43d9-9f48-1818d0bfe12e-kube-api-access-x42nq\") pod \"14e3d223-26f5-43d9-9f48-1818d0bfe12e\" (UID: \"14e3d223-26f5-43d9-9f48-1818d0bfe12e\") " Nov 03 02:32:58.327071 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.326662 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14e3d223-26f5-43d9-9f48-1818d0bfe12e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e3d223-26f5-43d9-9f48-1818d0bfe12e-plugins-config-volume\") pod \"14e3d223-26f5-43d9-9f48-1818d0bfe12e\" (UID: \"14e3d223-26f5-43d9-9f48-1818d0bfe12e\") " Nov 03 02:32:58.327071 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.326721 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9153ab8-8061-4275-b9ad-8aea6e949ab7-kube-api-access-zznjv\" (UniqueName: \"kubernetes.io/projected/d9153ab8-8061-4275-b9ad-8aea6e949ab7-kube-api-access-zznjv\") pod \"d9153ab8-8061-4275-b9ad-8aea6e949ab7\" (UID: \"d9153ab8-8061-4275-b9ad-8aea6e949ab7\") " Nov 03 02:32:58.327071 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.326754 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9153ab8-8061-4275-b9ad-8aea6e949ab7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9153ab8-8061-4275-b9ad-8aea6e949ab7-plugins-config-volume\") pod \"d9153ab8-8061-4275-b9ad-8aea6e949ab7\" (UID: \"d9153ab8-8061-4275-b9ad-8aea6e949ab7\") " Nov 03 02:32:58.327282 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.327141 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e3d223-26f5-43d9-9f48-1818d0bfe12e-plugins-config-volume" pod "14e3d223-26f5-43d9-9f48-1818d0bfe12e" (UID: "14e3d223-26f5-43d9-9f48-1818d0bfe12e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:32:58.327282 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.327165 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9153ab8-8061-4275-b9ad-8aea6e949ab7-plugins-config-volume" pod "d9153ab8-8061-4275-b9ad-8aea6e949ab7" (UID: "d9153ab8-8061-4275-b9ad-8aea6e949ab7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:32:58.327507 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.327487 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e3d223-26f5-43d9-9f48-1818d0bfe12e-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:58.327577 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.327512 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9153ab8-8061-4275-b9ad-8aea6e949ab7-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:58.329538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.329352 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e3d223-26f5-43d9-9f48-1818d0bfe12e-kube-api-access-x42nq" pod "14e3d223-26f5-43d9-9f48-1818d0bfe12e" (UID: "14e3d223-26f5-43d9-9f48-1818d0bfe12e"). InnerVolumeSpecName "kube-api-access-x42nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:32:58.329538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.329459 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9153ab8-8061-4275-b9ad-8aea6e949ab7-kube-api-access-zznjv" pod "d9153ab8-8061-4275-b9ad-8aea6e949ab7" (UID: "d9153ab8-8061-4275-b9ad-8aea6e949ab7"). InnerVolumeSpecName "kube-api-access-zznjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:32:58.448213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.446321 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x42nq\" (UniqueName: \"kubernetes.io/projected/14e3d223-26f5-43d9-9f48-1818d0bfe12e-kube-api-access-x42nq\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:58.448213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.446356 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zznjv\" (UniqueName: \"kubernetes.io/projected/d9153ab8-8061-4275-b9ad-8aea6e949ab7-kube-api-access-zznjv\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:58.493910 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.491138 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d372c46-0419-4d51-a4e6-0e651b656331" path="/var/lib/kubelet/pods/6d372c46-0419-4d51-a4e6-0e651b656331/volumes" Nov 03 02:32:58.602239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.602180 5463 generic.go:356] "Generic (PLEG): container finished" podID="d9153ab8-8061-4275-b9ad-8aea6e949ab7" containerID="36be79d2dd5147f499b052916c1061cb364d403a09f1bfcd46f0bb53c3283b02" exitCode=0 Nov 03 02:32:58.602239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.602222 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52" event={"ID":"d9153ab8-8061-4275-b9ad-8aea6e949ab7","Type":"ContainerDied","Data":"36be79d2dd5147f499b052916c1061cb364d403a09f1bfcd46f0bb53c3283b02"} Nov 03 02:32:58.602469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.602257 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52" Nov 03 02:32:58.602469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.602307 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52" event={"ID":"d9153ab8-8061-4275-b9ad-8aea6e949ab7","Type":"ContainerDied","Data":"5974272fbd7db50d6f4bc24dd509b37c14ca349e338260fb598baf3484b678d2"} Nov 03 02:32:58.602469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.602339 5463 scope.go:122] "RemoveContainer" containerID="36be79d2dd5147f499b052916c1061cb364d403a09f1bfcd46f0bb53c3283b02" Nov 03 02:32:58.603573 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.603518 5463 generic.go:356] "Generic (PLEG): container finished" podID="14e3d223-26f5-43d9-9f48-1818d0bfe12e" containerID="b08a9f479ac5100bb210533950056ac7412a36eb89cfdf4ffd0d695072d34b71" exitCode=0 Nov 03 02:32:58.603646 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.603578 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v" event={"ID":"14e3d223-26f5-43d9-9f48-1818d0bfe12e","Type":"ContainerDied","Data":"b08a9f479ac5100bb210533950056ac7412a36eb89cfdf4ffd0d695072d34b71"} Nov 03 02:32:58.603646 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.603608 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v" event={"ID":"14e3d223-26f5-43d9-9f48-1818d0bfe12e","Type":"ContainerDied","Data":"f130566df0a5ae6eaa3bb8f5e397d42a69348664e4ae4d0f560e4aa84950e9ed"} Nov 03 02:32:58.603727 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.603664 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v" Nov 03 02:32:58.616403 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.616384 5463 scope.go:122] "RemoveContainer" containerID="36be79d2dd5147f499b052916c1061cb364d403a09f1bfcd46f0bb53c3283b02" Nov 03 02:32:58.616710 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:32:58.616683 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36be79d2dd5147f499b052916c1061cb364d403a09f1bfcd46f0bb53c3283b02\": container with ID starting with 36be79d2dd5147f499b052916c1061cb364d403a09f1bfcd46f0bb53c3283b02 not found: ID does not exist" containerID="36be79d2dd5147f499b052916c1061cb364d403a09f1bfcd46f0bb53c3283b02" Nov 03 02:32:58.616811 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.616713 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36be79d2dd5147f499b052916c1061cb364d403a09f1bfcd46f0bb53c3283b02"} err="failed to get container status \"36be79d2dd5147f499b052916c1061cb364d403a09f1bfcd46f0bb53c3283b02\": rpc error: code = NotFound desc = could not find container \"36be79d2dd5147f499b052916c1061cb364d403a09f1bfcd46f0bb53c3283b02\": container with ID starting with 36be79d2dd5147f499b052916c1061cb364d403a09f1bfcd46f0bb53c3283b02 not found: ID does not exist" Nov 03 02:32:58.616811 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.616740 5463 scope.go:122] "RemoveContainer" containerID="b08a9f479ac5100bb210533950056ac7412a36eb89cfdf4ffd0d695072d34b71" Nov 03 02:32:58.627235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.627175 5463 scope.go:122] "RemoveContainer" containerID="b08a9f479ac5100bb210533950056ac7412a36eb89cfdf4ffd0d695072d34b71" Nov 03 02:32:58.627506 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:32:58.627487 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08a9f479ac5100bb210533950056ac7412a36eb89cfdf4ffd0d695072d34b71\": container with ID starting with b08a9f479ac5100bb210533950056ac7412a36eb89cfdf4ffd0d695072d34b71 not found: ID does not exist" containerID="b08a9f479ac5100bb210533950056ac7412a36eb89cfdf4ffd0d695072d34b71" Nov 03 02:32:58.627581 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.627514 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08a9f479ac5100bb210533950056ac7412a36eb89cfdf4ffd0d695072d34b71"} err="failed to get container status \"b08a9f479ac5100bb210533950056ac7412a36eb89cfdf4ffd0d695072d34b71\": rpc error: code = NotFound desc = could not find container \"b08a9f479ac5100bb210533950056ac7412a36eb89cfdf4ffd0d695072d34b71\": container with ID starting with b08a9f479ac5100bb210533950056ac7412a36eb89cfdf4ffd0d695072d34b71 not found: ID does not exist" Nov 03 02:32:58.768487 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.768405 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52"] Nov 03 02:32:58.771321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.771298 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb1940-765fd887b6-rhh52"] Nov 03 02:32:58.882656 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.882629 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v"] Nov 03 02:32:58.886310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:58.886285 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb1940-fbd8d4b75-sdj4v"] Nov 03 02:32:59.510448 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:59.510414 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:59.667795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:59.667764 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:59.992172 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:59.992135 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj"] Nov 03 02:32:59.992846 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:59.992827 5463 cpu_manager.go:414] "containerMap: removing container" podUID="6d372c46-0419-4d51-a4e6-0e651b656331" containerName="maas-api" Nov 03 02:32:59.992846 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:59.992846 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d372c46-0419-4d51-a4e6-0e651b656331" containerName="maas-api" Nov 03 02:32:59.993010 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:59.992864 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d9153ab8-8061-4275-b9ad-8aea6e949ab7" containerName="payload-processing" Nov 03 02:32:59.993010 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:59.992869 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9153ab8-8061-4275-b9ad-8aea6e949ab7" containerName="payload-processing" Nov 03 02:32:59.993010 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:59.992890 5463 cpu_manager.go:414] "containerMap: removing container" podUID="14e3d223-26f5-43d9-9f48-1818d0bfe12e" containerName="payload-pre-processing" Nov 03 02:32:59.993010 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:59.992896 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14e3d223-26f5-43d9-9f48-1818d0bfe12e" containerName="payload-pre-processing" Nov 03 02:32:59.993010 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:59.992992 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d372c46-0419-4d51-a4e6-0e651b656331" containerName="maas-api" Nov 03 02:32:59.993010 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:59.993000 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9153ab8-8061-4275-b9ad-8aea6e949ab7" containerName="payload-processing" Nov 03 02:32:59.993010 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:59.993009 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="14e3d223-26f5-43d9-9f48-1818d0bfe12e" containerName="payload-pre-processing" Nov 03 02:32:59.998118 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:32:59.998100 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.062228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.052538 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj"] Nov 03 02:33:00.065391 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.065363 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-workload-socket\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.065523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.065433 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04066f9c-931a-4c65-a714-57270768f0e5-istio-token\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.065585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.065532 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-workload-certs\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.065585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.065574 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04066f9c-931a-4c65-a714-57270768f0e5-istio-podinfo\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.065710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.065619 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-credential-socket\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.065710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.065659 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z89x\" (UniqueName: \"kubernetes.io/projected/04066f9c-931a-4c65-a714-57270768f0e5-kube-api-access-7z89x\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.065821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.065743 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-istio-envoy\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.065821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.065776 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-istio-data\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.065821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.065815 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04066f9c-931a-4c65-a714-57270768f0e5-istiod-ca-cert\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.070403 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.070384 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-965a07-openshift-default-dockercfg-65j56\"" Nov 03 02:33:00.097529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.097505 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:00.172212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.172031 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-workload-certs\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.172212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.172102 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04066f9c-931a-4c65-a714-57270768f0e5-istio-podinfo\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.172212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.172149 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-credential-socket\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.172212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.172203 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z89x\" (UniqueName: \"kubernetes.io/projected/04066f9c-931a-4c65-a714-57270768f0e5-kube-api-access-7z89x\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.172591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.172265 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-istio-envoy\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.172591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.172296 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-istio-data\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.172591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.172330 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04066f9c-931a-4c65-a714-57270768f0e5-istiod-ca-cert\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.172591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.172415 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-workload-socket\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.172591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.172492 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04066f9c-931a-4c65-a714-57270768f0e5-istio-token\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.192212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.173364 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-workload-certs\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.192212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.184886 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04066f9c-931a-4c65-a714-57270768f0e5-istiod-ca-cert\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.192212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.185158 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-istio-data\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.192212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.185463 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-workload-socket\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.192212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.185724 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-credential-socket\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.215212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.194534 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04066f9c-931a-4c65-a714-57270768f0e5-istio-podinfo\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.215212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.196815 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-istio-envoy\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.236281 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.232583 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04066f9c-931a-4c65-a714-57270768f0e5-istio-token\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.243458 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.243387 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z89x\" (UniqueName: \"kubernetes.io/projected/04066f9c-931a-4c65-a714-57270768f0e5-kube-api-access-7z89x\") pod \"e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.311811 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.311778 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:00.409022 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.408980 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:00.444453 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.444424 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14e3d223-26f5-43d9-9f48-1818d0bfe12e" path="/var/lib/kubelet/pods/14e3d223-26f5-43d9-9f48-1818d0bfe12e/volumes" Nov 03 02:33:00.444853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.444840 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9153ab8-8061-4275-b9ad-8aea6e949ab7" path="/var/lib/kubelet/pods/d9153ab8-8061-4275-b9ad-8aea6e949ab7/volumes" Nov 03 02:33:00.749149 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:33:00.749113 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04066f9c_931a_4c65_a714_57270768f0e5.slice/crio-57e878d91634251272fe505a1a3bdcc4acb05b23b8dbdc45181ffe1fcc7dab19 WatchSource:0}: Error finding container 57e878d91634251272fe505a1a3bdcc4acb05b23b8dbdc45181ffe1fcc7dab19: Status 404 returned error can't find the container with id 57e878d91634251272fe505a1a3bdcc4acb05b23b8dbdc45181ffe1fcc7dab19 Nov 03 02:33:00.751601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.751577 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj"] Nov 03 02:33:00.752004 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.751966 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:00.752078 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.752051 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:00.752142 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:00.752091 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:01.630000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:01.629953 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" event={"ID":"04066f9c-931a-4c65-a714-57270768f0e5","Type":"ContainerStarted","Data":"9188fa87fe3fa573942fbd56639310a8913d4380d4bd2f47c085360a8817826f"} Nov 03 02:33:01.630171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:01.630053 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" event={"ID":"04066f9c-931a-4c65-a714-57270768f0e5","Type":"ContainerStarted","Data":"57e878d91634251272fe505a1a3bdcc4acb05b23b8dbdc45181ffe1fcc7dab19"} Nov 03 02:33:01.802519 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:01.802456 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" podStartSLOduration=2.802437137 podStartE2EDuration="2.802437137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:01.79911487 +0000 UTC m=+2463.483224036" watchObservedRunningTime="2026-11-03 02:33:01.802437137 +0000 UTC m=+2463.486546300" Nov 03 02:33:02.312129 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:02.312089 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:02.317468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:02.317445 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:02.636500 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:02.636419 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:02.637445 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:02.637419 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:33:04.343654 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:04.343616 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:04.555121 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:04.555087 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:04.742360 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:04.742325 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:04.934226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:04.927859 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:05.241418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:05.241375 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:05.533469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:05.533366 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:06.739216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:06.724671 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:07.639214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:07.639107 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:07.823448 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:07.823407 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:08.228240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:08.223850 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:08.393216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:08.392844 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:08.885245 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:08.885210 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl"] Nov 03 02:33:08.890946 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:08.890923 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl" Nov 03 02:33:08.952331 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:08.952291 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl"] Nov 03 02:33:08.986102 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:08.986030 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mxlj\" (UniqueName: \"kubernetes.io/projected/f4cd396d-6d5a-4b80-ad42-48646deb9e26-kube-api-access-8mxlj\") pod \"payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl\" (UID: \"f4cd396d-6d5a-4b80-ad42-48646deb9e26\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl" Nov 03 02:33:08.986301 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:08.986237 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4cd396d-6d5a-4b80-ad42-48646deb9e26-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl\" (UID: \"f4cd396d-6d5a-4b80-ad42-48646deb9e26\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl" Nov 03 02:33:08.988784 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:08.988747 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-965a07\"" Nov 03 02:33:08.988944 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:08.988838 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-965a07-dockercfg-kjxms\"" Nov 03 02:33:09.087000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:09.086969 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mxlj\" (UniqueName: \"kubernetes.io/projected/f4cd396d-6d5a-4b80-ad42-48646deb9e26-kube-api-access-8mxlj\") pod \"payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl\" (UID: \"f4cd396d-6d5a-4b80-ad42-48646deb9e26\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl" Nov 03 02:33:09.087207 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:09.087062 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4cd396d-6d5a-4b80-ad42-48646deb9e26-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl\" (UID: \"f4cd396d-6d5a-4b80-ad42-48646deb9e26\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl" Nov 03 02:33:09.087753 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:09.087732 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4cd396d-6d5a-4b80-ad42-48646deb9e26-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl\" (UID: \"f4cd396d-6d5a-4b80-ad42-48646deb9e26\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl" Nov 03 02:33:09.147775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:09.147746 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mxlj\" (UniqueName: \"kubernetes.io/projected/f4cd396d-6d5a-4b80-ad42-48646deb9e26-kube-api-access-8mxlj\") pod \"payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl\" (UID: \"f4cd396d-6d5a-4b80-ad42-48646deb9e26\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl" Nov 03 02:33:09.205816 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:09.205787 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl" Nov 03 02:33:09.236341 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:09.236304 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:09.593837 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:33:09.593796 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cd396d_6d5a_4b80_ad42_48646deb9e26.slice/crio-5415a85dec97c05460bf35c1f464fe90713cbdd228037a5a14e14f042ba940a4 WatchSource:0}: Error finding container 5415a85dec97c05460bf35c1f464fe90713cbdd228037a5a14e14f042ba940a4: Status 404 returned error can't find the container with id 5415a85dec97c05460bf35c1f464fe90713cbdd228037a5a14e14f042ba940a4 Nov 03 02:33:09.596187 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:09.596163 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl"] Nov 03 02:33:09.708251 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:09.708224 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj"] Nov 03 02:33:09.708486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:09.708416 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" podUID="45226bbf-4673-4108-bb2d-962a40340fb6" containerName="istio-proxy" containerID="cri-o://6b423d45bd051448a81ba53a700b5c4dd74b47522a48e4b28d445da3db58cde3" gracePeriod=30 Nov 03 02:33:09.737448 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:09.737426 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:09.741254 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:09.741233 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl" event={"ID":"f4cd396d-6d5a-4b80-ad42-48646deb9e26","Type":"ContainerStarted","Data":"80043db282d2ade8e2e834587f37e3569ccd2281d3eeab54218404163956033e"} Nov 03 02:33:09.741342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:09.741276 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl" event={"ID":"f4cd396d-6d5a-4b80-ad42-48646deb9e26","Type":"ContainerStarted","Data":"5415a85dec97c05460bf35c1f464fe90713cbdd228037a5a14e14f042ba940a4"} Nov 03 02:33:09.741342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:09.741302 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl" Nov 03 02:33:09.953310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:09.953262 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl" podStartSLOduration=1.9532492559999999 podStartE2EDuration="1.953249256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:09.950013781 +0000 UTC m=+2471.634122933" watchObservedRunningTime="2026-11-03 02:33:09.953249256 +0000 UTC m=+2471.637358410" Nov 03 02:33:10.560309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:10.560276 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:10.950075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:10.950041 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:11.118262 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:11.118227 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:11.297872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:11.297764 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:12.175208 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:12.175133 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:12.857521 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:12.857485 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr"] Nov 03 02:33:12.862378 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:12.862352 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr" Nov 03 02:33:12.912806 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:12.912772 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr"] Nov 03 02:33:12.930815 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:12.930785 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg52r\" (UniqueName: \"kubernetes.io/projected/c8e096d0-d67b-452a-bd08-870c210c52c5-kube-api-access-zg52r\") pod \"maas-api-revoke-sub-2a3eeb8d-4qsbr\" (UID: \"c8e096d0-d67b-452a-bd08-870c210c52c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr" Nov 03 02:33:12.930997 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:12.930963 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e096d0-d67b-452a-bd08-870c210c52c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a3eeb8d-4qsbr\" (UID: \"c8e096d0-d67b-452a-bd08-870c210c52c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr" Nov 03 02:33:12.931161 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:12.931133 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e096d0-d67b-452a-bd08-870c210c52c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a3eeb8d-4qsbr\" (UID: \"c8e096d0-d67b-452a-bd08-870c210c52c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr" Nov 03 02:33:12.959680 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:12.959653 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:33:12.959680 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:12.959670 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:33:12.959922 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:12.959718 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:33:12.963788 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:12.963768 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:33:13.009104 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:33:13.009071 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:33:13.032820 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:13.032797 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg52r\" (UniqueName: \"kubernetes.io/projected/c8e096d0-d67b-452a-bd08-870c210c52c5-kube-api-access-zg52r\") pod \"maas-api-revoke-sub-2a3eeb8d-4qsbr\" (UID: \"c8e096d0-d67b-452a-bd08-870c210c52c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr" Nov 03 02:33:13.032953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:13.032901 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e096d0-d67b-452a-bd08-870c210c52c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a3eeb8d-4qsbr\" (UID: \"c8e096d0-d67b-452a-bd08-870c210c52c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr" Nov 03 02:33:13.033008 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:13.032993 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e096d0-d67b-452a-bd08-870c210c52c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a3eeb8d-4qsbr\" (UID: \"c8e096d0-d67b-452a-bd08-870c210c52c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr" Nov 03 02:33:13.033693 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:13.033669 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e096d0-d67b-452a-bd08-870c210c52c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a3eeb8d-4qsbr\" (UID: \"c8e096d0-d67b-452a-bd08-870c210c52c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr" Nov 03 02:33:13.035131 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:13.035113 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e096d0-d67b-452a-bd08-870c210c52c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a3eeb8d-4qsbr\" (UID: \"c8e096d0-d67b-452a-bd08-870c210c52c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr" Nov 03 02:33:13.090185 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:13.090154 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg52r\" (UniqueName: \"kubernetes.io/projected/c8e096d0-d67b-452a-bd08-870c210c52c5-kube-api-access-zg52r\") pod \"maas-api-revoke-sub-2a3eeb8d-4qsbr\" (UID: \"c8e096d0-d67b-452a-bd08-870c210c52c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr" Nov 03 02:33:13.190591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:13.190558 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr" Nov 03 02:33:13.562286 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:33:13.562251 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e096d0_d67b_452a_bd08_870c210c52c5.slice/crio-5d1e41a5af61c3261f3c5441c54cd959eef17b5291119de7eea0ae7ada7af6ac WatchSource:0}: Error finding container 5d1e41a5af61c3261f3c5441c54cd959eef17b5291119de7eea0ae7ada7af6ac: Status 404 returned error can't find the container with id 5d1e41a5af61c3261f3c5441c54cd959eef17b5291119de7eea0ae7ada7af6ac Nov 03 02:33:13.566157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:13.566134 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr"] Nov 03 02:33:13.774061 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:13.774028 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr" event={"ID":"c8e096d0-d67b-452a-bd08-870c210c52c5","Type":"ContainerStarted","Data":"5d1e41a5af61c3261f3c5441c54cd959eef17b5291119de7eea0ae7ada7af6ac"} Nov 03 02:33:14.155111 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.155071 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:14.322911 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.322835 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:14.543859 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.543832 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns"] Nov 03 02:33:14.547847 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.547831 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns" Nov 03 02:33:14.602887 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.602812 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns"] Nov 03 02:33:14.655274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.652457 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-3a4683\"" Nov 03 02:33:14.655274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.652631 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-3a4683-dockercfg-vfrd8\"" Nov 03 02:33:14.665145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.665107 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9plnx\" (UniqueName: \"kubernetes.io/projected/d683621e-ee05-4efe-9d28-49d1f085d135-kube-api-access-9plnx\") pod \"payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns\" (UID: \"d683621e-ee05-4efe-9d28-49d1f085d135\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns" Nov 03 02:33:14.665396 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.665172 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d683621e-ee05-4efe-9d28-49d1f085d135-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns\" (UID: \"d683621e-ee05-4efe-9d28-49d1f085d135\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns" Nov 03 02:33:14.698032 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.697996 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8"] Nov 03 02:33:14.703062 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.702206 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8" Nov 03 02:33:14.757897 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.757861 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8"] Nov 03 02:33:14.766441 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.766415 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9plnx\" (UniqueName: \"kubernetes.io/projected/d683621e-ee05-4efe-9d28-49d1f085d135-kube-api-access-9plnx\") pod \"payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns\" (UID: \"d683621e-ee05-4efe-9d28-49d1f085d135\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns" Nov 03 02:33:14.766582 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.766474 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d683621e-ee05-4efe-9d28-49d1f085d135-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns\" (UID: \"d683621e-ee05-4efe-9d28-49d1f085d135\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns" Nov 03 02:33:14.766662 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.766609 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6e7a0f-9fd7-4115-9c62-caed63f180de-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8\" (UID: \"da6e7a0f-9fd7-4115-9c62-caed63f180de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8" Nov 03 02:33:14.766722 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.766662 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4vsq\" (UniqueName: \"kubernetes.io/projected/da6e7a0f-9fd7-4115-9c62-caed63f180de-kube-api-access-f4vsq\") pod \"payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8\" (UID: \"da6e7a0f-9fd7-4115-9c62-caed63f180de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8" Nov 03 02:33:14.767268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.767184 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d683621e-ee05-4efe-9d28-49d1f085d135-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns\" (UID: \"d683621e-ee05-4efe-9d28-49d1f085d135\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns" Nov 03 02:33:14.784267 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.784243 5463 generic.go:356] "Generic (PLEG): container finished" podID="c8e096d0-d67b-452a-bd08-870c210c52c5" containerID="ef3e2d14c27ab232bf2fe68c4cca7898221fc18170933346aeed63184bf9526a" exitCode=0 Nov 03 02:33:14.784393 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.784329 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr" event={"ID":"c8e096d0-d67b-452a-bd08-870c210c52c5","Type":"ContainerDied","Data":"ef3e2d14c27ab232bf2fe68c4cca7898221fc18170933346aeed63184bf9526a"} Nov 03 02:33:14.824590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.824565 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9plnx\" (UniqueName: \"kubernetes.io/projected/d683621e-ee05-4efe-9d28-49d1f085d135-kube-api-access-9plnx\") pod \"payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns\" (UID: \"d683621e-ee05-4efe-9d28-49d1f085d135\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns" Nov 03 02:33:14.861723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.861654 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns" Nov 03 02:33:14.867459 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.867433 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6e7a0f-9fd7-4115-9c62-caed63f180de-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8\" (UID: \"da6e7a0f-9fd7-4115-9c62-caed63f180de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8" Nov 03 02:33:14.867603 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.867485 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4vsq\" (UniqueName: \"kubernetes.io/projected/da6e7a0f-9fd7-4115-9c62-caed63f180de-kube-api-access-f4vsq\") pod \"payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8\" (UID: \"da6e7a0f-9fd7-4115-9c62-caed63f180de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8" Nov 03 02:33:14.868126 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.868102 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6e7a0f-9fd7-4115-9c62-caed63f180de-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8\" (UID: \"da6e7a0f-9fd7-4115-9c62-caed63f180de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8" Nov 03 02:33:14.928127 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:14.928075 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4vsq\" (UniqueName: \"kubernetes.io/projected/da6e7a0f-9fd7-4115-9c62-caed63f180de-kube-api-access-f4vsq\") pod \"payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8\" (UID: \"da6e7a0f-9fd7-4115-9c62-caed63f180de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8" Nov 03 02:33:15.005786 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.005761 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:33:15.022888 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.022865 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8" Nov 03 02:33:15.070142 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.070113 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45226bbf-4673-4108-bb2d-962a40340fb6-istio-token\" (UniqueName: \"kubernetes.io/projected/45226bbf-4673-4108-bb2d-962a40340fb6-istio-token\") pod \"45226bbf-4673-4108-bb2d-962a40340fb6\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " Nov 03 02:33:15.070306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.070157 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45226bbf-4673-4108-bb2d-962a40340fb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45226bbf-4673-4108-bb2d-962a40340fb6-istio-podinfo\") pod \"45226bbf-4673-4108-bb2d-962a40340fb6\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " Nov 03 02:33:15.070306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.070214 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-workload-socket\") pod \"45226bbf-4673-4108-bb2d-962a40340fb6\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " Nov 03 02:33:15.070306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.070269 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45226bbf-4673-4108-bb2d-962a40340fb6-kube-api-access-p298s\" (UniqueName: \"kubernetes.io/projected/45226bbf-4673-4108-bb2d-962a40340fb6-kube-api-access-p298s\") pod \"45226bbf-4673-4108-bb2d-962a40340fb6\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " Nov 03 02:33:15.070537 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.070370 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-credential-socket\") pod \"45226bbf-4673-4108-bb2d-962a40340fb6\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " Nov 03 02:33:15.070537 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.070429 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-istio-data\") pod \"45226bbf-4673-4108-bb2d-962a40340fb6\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " Nov 03 02:33:15.070537 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.070466 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-workload-certs\") pod \"45226bbf-4673-4108-bb2d-962a40340fb6\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " Nov 03 02:33:15.070537 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.070528 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45226bbf-4673-4108-bb2d-962a40340fb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45226bbf-4673-4108-bb2d-962a40340fb6-istiod-ca-cert\") pod \"45226bbf-4673-4108-bb2d-962a40340fb6\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " Nov 03 02:33:15.070766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.070575 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-istio-envoy\") pod \"45226bbf-4673-4108-bb2d-962a40340fb6\" (UID: \"45226bbf-4673-4108-bb2d-962a40340fb6\") " Nov 03 02:33:15.070766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.070639 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-workload-socket" pod "45226bbf-4673-4108-bb2d-962a40340fb6" (UID: "45226bbf-4673-4108-bb2d-962a40340fb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:15.070766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.070666 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-credential-socket" pod "45226bbf-4673-4108-bb2d-962a40340fb6" (UID: "45226bbf-4673-4108-bb2d-962a40340fb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:15.070967 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.070944 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-istio-data" pod "45226bbf-4673-4108-bb2d-962a40340fb6" (UID: "45226bbf-4673-4108-bb2d-962a40340fb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:15.071206 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.071169 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-workload-certs" pod "45226bbf-4673-4108-bb2d-962a40340fb6" (UID: "45226bbf-4673-4108-bb2d-962a40340fb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:15.071364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.071344 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45226bbf-4673-4108-bb2d-962a40340fb6-istiod-ca-cert" pod "45226bbf-4673-4108-bb2d-962a40340fb6" (UID: "45226bbf-4673-4108-bb2d-962a40340fb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:33:15.071693 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.071669 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:15.071801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.071699 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:15.071801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.071715 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:15.071801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.071730 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:15.071801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.071744 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45226bbf-4673-4108-bb2d-962a40340fb6-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:15.074129 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.074105 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45226bbf-4673-4108-bb2d-962a40340fb6-istio-podinfo" pod "45226bbf-4673-4108-bb2d-962a40340fb6" (UID: "45226bbf-4673-4108-bb2d-962a40340fb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:33:15.074129 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.074123 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45226bbf-4673-4108-bb2d-962a40340fb6-istio-token" pod "45226bbf-4673-4108-bb2d-962a40340fb6" (UID: "45226bbf-4673-4108-bb2d-962a40340fb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:15.077956 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.077921 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-istio-envoy" pod "45226bbf-4673-4108-bb2d-962a40340fb6" (UID: "45226bbf-4673-4108-bb2d-962a40340fb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:15.081243 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.079412 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45226bbf-4673-4108-bb2d-962a40340fb6-kube-api-access-p298s" pod "45226bbf-4673-4108-bb2d-962a40340fb6" (UID: "45226bbf-4673-4108-bb2d-962a40340fb6"). InnerVolumeSpecName "kube-api-access-p298s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:15.173168 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.173125 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45226bbf-4673-4108-bb2d-962a40340fb6-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:15.173396 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.173181 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p298s\" (UniqueName: \"kubernetes.io/projected/45226bbf-4673-4108-bb2d-962a40340fb6-kube-api-access-p298s\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:15.173396 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.173241 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45226bbf-4673-4108-bb2d-962a40340fb6-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:15.173396 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.173256 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45226bbf-4673-4108-bb2d-962a40340fb6-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:15.258003 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:33:15.257955 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd683621e_ee05_4efe_9d28_49d1f085d135.slice/crio-910e33ff49ae8e4ee1c2c404dd02e185a7a170e9a7f394192afbd41ed25392b6 WatchSource:0}: Error finding container 910e33ff49ae8e4ee1c2c404dd02e185a7a170e9a7f394192afbd41ed25392b6: Status 404 returned error can't find the container with id 910e33ff49ae8e4ee1c2c404dd02e185a7a170e9a7f394192afbd41ed25392b6 Nov 03 02:33:15.262335 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.262312 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns"] Nov 03 02:33:15.390497 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:33:15.390456 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6e7a0f_9fd7_4115_9c62_caed63f180de.slice/crio-50eaead815e755bcecc0bf1af23ec3159e6d692a1b3dc591eb0f4f5c3df77b37 WatchSource:0}: Error finding container 50eaead815e755bcecc0bf1af23ec3159e6d692a1b3dc591eb0f4f5c3df77b37: Status 404 returned error can't find the container with id 50eaead815e755bcecc0bf1af23ec3159e6d692a1b3dc591eb0f4f5c3df77b37 Nov 03 02:33:15.393503 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.393475 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8"] Nov 03 02:33:15.798075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.797984 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8" event={"ID":"da6e7a0f-9fd7-4115-9c62-caed63f180de","Type":"ContainerStarted","Data":"a67c47e441a2d717ba1dcd75ca3ed7911d700e8c1f0c080a2028e1b56370804d"} Nov 03 02:33:15.798344 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.798325 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8" Nov 03 02:33:15.798463 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.798441 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8" event={"ID":"da6e7a0f-9fd7-4115-9c62-caed63f180de","Type":"ContainerStarted","Data":"50eaead815e755bcecc0bf1af23ec3159e6d692a1b3dc591eb0f4f5c3df77b37"} Nov 03 02:33:15.800196 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.800169 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns" event={"ID":"d683621e-ee05-4efe-9d28-49d1f085d135","Type":"ContainerStarted","Data":"b409b192f53cda255baf81a70689f4c79362d44c78d9e2e563611fc57e524df3"} Nov 03 02:33:15.800327 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.800228 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns" event={"ID":"d683621e-ee05-4efe-9d28-49d1f085d135","Type":"ContainerStarted","Data":"910e33ff49ae8e4ee1c2c404dd02e185a7a170e9a7f394192afbd41ed25392b6"} Nov 03 02:33:15.800327 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.800257 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns" Nov 03 02:33:15.802171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.802149 5463 generic.go:356] "Generic (PLEG): container finished" podID="45226bbf-4673-4108-bb2d-962a40340fb6" containerID="6b423d45bd051448a81ba53a700b5c4dd74b47522a48e4b28d445da3db58cde3" exitCode=0 Nov 03 02:33:15.802321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.802212 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" Nov 03 02:33:15.802321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.802219 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" event={"ID":"45226bbf-4673-4108-bb2d-962a40340fb6","Type":"ContainerDied","Data":"6b423d45bd051448a81ba53a700b5c4dd74b47522a48e4b28d445da3db58cde3"} Nov 03 02:33:15.802321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.802268 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj" event={"ID":"45226bbf-4673-4108-bb2d-962a40340fb6","Type":"ContainerDied","Data":"39dc40e89441ad3476fef9ff5fca7466f1b6cc2bd353a1a02d0b3428eb3f07f1"} Nov 03 02:33:15.802321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.802297 5463 scope.go:122] "RemoveContainer" containerID="6b423d45bd051448a81ba53a700b5c4dd74b47522a48e4b28d445da3db58cde3" Nov 03 02:33:15.816810 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.816793 5463 scope.go:122] "RemoveContainer" containerID="37642d83a1157566381133b6f350635afe59ea09ffc227a8193d3cb5f95acf75" Nov 03 02:33:15.827231 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.826995 5463 scope.go:122] "RemoveContainer" containerID="6b423d45bd051448a81ba53a700b5c4dd74b47522a48e4b28d445da3db58cde3" Nov 03 02:33:15.827393 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:33:15.827365 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b423d45bd051448a81ba53a700b5c4dd74b47522a48e4b28d445da3db58cde3\": container with ID starting with 6b423d45bd051448a81ba53a700b5c4dd74b47522a48e4b28d445da3db58cde3 not found: ID does not exist" containerID="6b423d45bd051448a81ba53a700b5c4dd74b47522a48e4b28d445da3db58cde3" Nov 03 02:33:15.827469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.827404 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b423d45bd051448a81ba53a700b5c4dd74b47522a48e4b28d445da3db58cde3"} err="failed to get container status \"6b423d45bd051448a81ba53a700b5c4dd74b47522a48e4b28d445da3db58cde3\": rpc error: code = NotFound desc = could not find container \"6b423d45bd051448a81ba53a700b5c4dd74b47522a48e4b28d445da3db58cde3\": container with ID starting with 6b423d45bd051448a81ba53a700b5c4dd74b47522a48e4b28d445da3db58cde3 not found: ID does not exist" Nov 03 02:33:15.827469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.827440 5463 scope.go:122] "RemoveContainer" containerID="37642d83a1157566381133b6f350635afe59ea09ffc227a8193d3cb5f95acf75" Nov 03 02:33:15.827762 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:33:15.827723 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37642d83a1157566381133b6f350635afe59ea09ffc227a8193d3cb5f95acf75\": container with ID starting with 37642d83a1157566381133b6f350635afe59ea09ffc227a8193d3cb5f95acf75 not found: ID does not exist" containerID="37642d83a1157566381133b6f350635afe59ea09ffc227a8193d3cb5f95acf75" Nov 03 02:33:15.827856 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.827761 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37642d83a1157566381133b6f350635afe59ea09ffc227a8193d3cb5f95acf75"} err="failed to get container status \"37642d83a1157566381133b6f350635afe59ea09ffc227a8193d3cb5f95acf75\": rpc error: code = NotFound desc = could not find container \"37642d83a1157566381133b6f350635afe59ea09ffc227a8193d3cb5f95acf75\": container with ID starting with 37642d83a1157566381133b6f350635afe59ea09ffc227a8193d3cb5f95acf75 not found: ID does not exist" Nov 03 02:33:15.915099 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.915045 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8" podStartSLOduration=1.91502833 podStartE2EDuration="1.91502833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:15.912090337 +0000 UTC m=+2477.596199494" watchObservedRunningTime="2026-11-03 02:33:15.91502833 +0000 UTC m=+2477.599137486" Nov 03 02:33:15.961971 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.961943 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr" Nov 03 02:33:15.983285 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.983257 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8e096d0-d67b-452a-bd08-870c210c52c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e096d0-d67b-452a-bd08-870c210c52c5-maas-api-xks-ca\") pod \"c8e096d0-d67b-452a-bd08-870c210c52c5\" (UID: \"c8e096d0-d67b-452a-bd08-870c210c52c5\") " Nov 03 02:33:15.983399 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.983310 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e096d0-d67b-452a-bd08-870c210c52c5-kube-api-access-zg52r\" (UniqueName: \"kubernetes.io/projected/c8e096d0-d67b-452a-bd08-870c210c52c5-kube-api-access-zg52r\") pod \"c8e096d0-d67b-452a-bd08-870c210c52c5\" (UID: \"c8e096d0-d67b-452a-bd08-870c210c52c5\") " Nov 03 02:33:15.983459 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.983450 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8e096d0-d67b-452a-bd08-870c210c52c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e096d0-d67b-452a-bd08-870c210c52c5-maas-api-openshift-ca\") pod \"c8e096d0-d67b-452a-bd08-870c210c52c5\" (UID: \"c8e096d0-d67b-452a-bd08-870c210c52c5\") " Nov 03 02:33:15.983841 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.983801 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e096d0-d67b-452a-bd08-870c210c52c5-maas-api-openshift-ca" pod "c8e096d0-d67b-452a-bd08-870c210c52c5" (UID: "c8e096d0-d67b-452a-bd08-870c210c52c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:33:15.985418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.985394 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8e096d0-d67b-452a-bd08-870c210c52c5-maas-api-xks-ca" pod "c8e096d0-d67b-452a-bd08-870c210c52c5" (UID: "c8e096d0-d67b-452a-bd08-870c210c52c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:33:15.985418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:15.985407 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e096d0-d67b-452a-bd08-870c210c52c5-kube-api-access-zg52r" pod "c8e096d0-d67b-452a-bd08-870c210c52c5" (UID: "c8e096d0-d67b-452a-bd08-870c210c52c5"). InnerVolumeSpecName "kube-api-access-zg52r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:16.030314 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:16.030259 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns" podStartSLOduration=2.030242425 podStartE2EDuration="2.030242425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:16.026343094 +0000 UTC m=+2477.710452249" watchObservedRunningTime="2026-11-03 02:33:16.030242425 +0000 UTC m=+2477.714351581" Nov 03 02:33:16.084922 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:16.084813 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e096d0-d67b-452a-bd08-870c210c52c5-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:16.084922 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:16.084849 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg52r\" (UniqueName: \"kubernetes.io/projected/c8e096d0-d67b-452a-bd08-870c210c52c5-kube-api-access-zg52r\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:16.084922 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:16.084874 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e096d0-d67b-452a-bd08-870c210c52c5-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:16.141731 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:16.141699 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj"] Nov 03 02:33:16.146675 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:16.146651 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-fb1940-openshift-default-65d5f5787-vbbmj"] Nov 03 02:33:16.443482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:16.443447 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45226bbf-4673-4108-bb2d-962a40340fb6" path="/var/lib/kubelet/pods/45226bbf-4673-4108-bb2d-962a40340fb6/volumes" Nov 03 02:33:16.461991 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:16.461949 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:16.809993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:16.809920 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr" Nov 03 02:33:16.809993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:16.809948 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr" event={"ID":"c8e096d0-d67b-452a-bd08-870c210c52c5","Type":"ContainerDied","Data":"5d1e41a5af61c3261f3c5441c54cd959eef17b5291119de7eea0ae7ada7af6ac"} Nov 03 02:33:16.810224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:16.810001 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d1e41a5af61c3261f3c5441c54cd959eef17b5291119de7eea0ae7ada7af6ac" Nov 03 02:33:16.910092 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:16.910068 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:17.575476 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:17.575426 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:20.750859 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:20.750832 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl" Nov 03 02:33:22.623643 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:22.623608 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:23.970897 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:23.970825 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:24.133606 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.133571 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:24.481422 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.481387 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp"] Nov 03 02:33:24.482386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.482366 5463 cpu_manager.go:414] "containerMap: removing container" podUID="45226bbf-4673-4108-bb2d-962a40340fb6" containerName="istio-proxy" Nov 03 02:33:24.482477 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.482388 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45226bbf-4673-4108-bb2d-962a40340fb6" containerName="istio-proxy" Nov 03 02:33:24.482477 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.482438 5463 cpu_manager.go:414] "containerMap: removing container" podUID="c8e096d0-d67b-452a-bd08-870c210c52c5" containerName="revoke-keys" Nov 03 02:33:24.482477 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.482445 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e096d0-d67b-452a-bd08-870c210c52c5" containerName="revoke-keys" Nov 03 02:33:24.482624 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.482613 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="45226bbf-4673-4108-bb2d-962a40340fb6" containerName="istio-proxy" Nov 03 02:33:24.482664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.482640 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e096d0-d67b-452a-bd08-870c210c52c5" containerName="revoke-keys" Nov 03 02:33:24.487558 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.487536 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.579748 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.579714 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp"] Nov 03 02:33:24.590971 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.590934 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-9aebe8-openshift-default-dockercfg-s678n\"" Nov 03 02:33:24.591657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.591630 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k55xx\" (UniqueName: \"kubernetes.io/projected/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-kube-api-access-k55xx\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.591657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.591634 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:24.591849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.591757 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-envoy\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.591849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.591800 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-workload-socket\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.591849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.591830 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-workload-certs\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.591849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.591847 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-data\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.592069 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.591889 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istiod-ca-cert\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.592069 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.591909 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-podinfo\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.592069 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.591986 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-token\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.592069 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.592006 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-credential-socket\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.706723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.699085 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k55xx\" (UniqueName: \"kubernetes.io/projected/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-kube-api-access-k55xx\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.706723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.699228 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-envoy\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.706723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.699276 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-workload-socket\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.706723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.699311 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-workload-certs\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.706723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.699337 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-data\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.706723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.699394 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istiod-ca-cert\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.706723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.699424 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-podinfo\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.706723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.699519 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-token\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.706723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.699550 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-credential-socket\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.706723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.700087 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-credential-socket\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.706723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.703215 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-envoy\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.706723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.703434 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-data\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.706723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.703697 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-workload-socket\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.706723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.703886 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istiod-ca-cert\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.706723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.703940 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-workload-certs\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.706723 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.706656 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-podinfo\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.776260 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.772786 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-token\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.791264 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.791226 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg"] Nov 03 02:33:24.794202 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.792322 5463 cpu_manager.go:414] "containerMap: removing container" podUID="45226bbf-4673-4108-bb2d-962a40340fb6" containerName="istio-proxy" Nov 03 02:33:24.794202 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.792345 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45226bbf-4673-4108-bb2d-962a40340fb6" containerName="istio-proxy" Nov 03 02:33:24.794202 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.792514 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="45226bbf-4673-4108-bb2d-962a40340fb6" containerName="istio-proxy" Nov 03 02:33:24.819426 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.810679 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:24.819426 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.812097 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k55xx\" (UniqueName: \"kubernetes.io/projected/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-kube-api-access-k55xx\") pod \"e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:24.863933 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.863874 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg"] Nov 03 02:33:24.897446 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.889368 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-f735f4-openshift-default-dockercfg-4w55v\"" Nov 03 02:33:24.897446 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.895435 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:24.920279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.920243 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:24.922021 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.921992 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:24.922304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.922284 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:24.922490 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.922469 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:24.922685 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.922665 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:24.922887 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.922871 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:24.923066 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.923048 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:24.923249 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.923225 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2slns\" (UniqueName: \"kubernetes.io/projected/2d2157b2-209f-4ed7-9aab-a79fbe023d82-kube-api-access-2slns\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:24.923407 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:24.923390 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:25.025291 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.025239 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:25.025762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.025326 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:25.025762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.025355 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:25.025762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.025380 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:25.025762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.025461 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:25.025762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.025516 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:25.025762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.025568 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2slns\" (UniqueName: \"kubernetes.io/projected/2d2157b2-209f-4ed7-9aab-a79fbe023d82-kube-api-access-2slns\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:25.025762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.025600 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:25.025762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.025668 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:25.026233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.025914 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:25.026233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.026179 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:25.026396 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.026252 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:25.026396 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.026324 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:25.026521 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.026465 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:25.027659 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.027638 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:25.028151 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.028127 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:25.067863 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.067807 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:25.083641 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.083614 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:25.083771 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.083752 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2slns\" (UniqueName: \"kubernetes.io/projected/2d2157b2-209f-4ed7-9aab-a79fbe023d82-kube-api-access-2slns\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:25.102826 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.102801 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:25.230546 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.230514 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:25.241514 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.241484 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:25.551711 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:33:25.551676 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ddcf4ff_6e4f_45cd_81a8_8b283429291a.slice/crio-87903f50bf49cabecece6e47c78efb206582fb0eba06ff38e81d5e2d51fbef2a WatchSource:0}: Error finding container 87903f50bf49cabecece6e47c78efb206582fb0eba06ff38e81d5e2d51fbef2a: Status 404 returned error can't find the container with id 87903f50bf49cabecece6e47c78efb206582fb0eba06ff38e81d5e2d51fbef2a Nov 03 02:33:25.553514 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.553480 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp"] Nov 03 02:33:25.555075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.555008 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:25.555163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.555106 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:25.555163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.555150 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:25.660955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.660908 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg"] Nov 03 02:33:25.661085 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.661033 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:25.661151 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.661097 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:25.661151 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.661134 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:25.935713 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.935669 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" event={"ID":"5ddcf4ff-6e4f-45cd-81a8-8b283429291a","Type":"ContainerStarted","Data":"8f57f4bda498318d5b27f65f2842ab61418b7f4cb5577faa26326b5bc9ee2154"} Nov 03 02:33:25.935901 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.935731 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" event={"ID":"5ddcf4ff-6e4f-45cd-81a8-8b283429291a","Type":"ContainerStarted","Data":"87903f50bf49cabecece6e47c78efb206582fb0eba06ff38e81d5e2d51fbef2a"} Nov 03 02:33:25.937432 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.937387 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" event={"ID":"2d2157b2-209f-4ed7-9aab-a79fbe023d82","Type":"ContainerStarted","Data":"92bf2fd408a99e0a6e1de092dee3d681167da3227d30119d33c6a565c74042b6"} Nov 03 02:33:25.937554 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:25.937458 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" event={"ID":"2d2157b2-209f-4ed7-9aab-a79fbe023d82","Type":"ContainerStarted","Data":"3a085d3d9315b5ef6b1e4372dcd0cbb1597f06efec3087ed46b6291b6b23abbe"} Nov 03 02:33:26.056883 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:26.056822 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" podStartSLOduration=2.056803421 podStartE2EDuration="2.056803421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:26.054243645 +0000 UTC m=+2487.738352802" watchObservedRunningTime="2026-11-03 02:33:26.056803421 +0000 UTC m=+2487.740912579" Nov 03 02:33:26.103065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:26.103028 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:26.109113 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:26.109086 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:26.231536 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:26.231455 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:26.236702 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:26.236675 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:26.300278 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:26.300232 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" podStartSLOduration=2.300219174 podStartE2EDuration="2.300219174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:26.180418813 +0000 UTC m=+2487.864527971" watchObservedRunningTime="2026-11-03 02:33:26.300219174 +0000 UTC m=+2487.984328388" Nov 03 02:33:26.814678 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:26.814648 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8" Nov 03 02:33:26.814879 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:26.814720 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns" Nov 03 02:33:26.947172 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:26.947144 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:26.947680 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:26.947652 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:26.947807 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:26.947721 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:33:26.948481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:26.948461 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:33:27.114687 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.114600 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj"] Nov 03 02:33:27.119134 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.119110 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.197212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.196813 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj"] Nov 03 02:33:27.269781 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.268393 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.269781 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.268582 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.269781 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.268721 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkgkd\" (UniqueName: \"kubernetes.io/projected/ba4b9eb7-fc25-451c-9e9f-aae89573754d-kube-api-access-nkgkd\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.269781 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.268809 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.269781 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.268866 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.269781 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.268941 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.269781 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.269036 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.269781 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.269109 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.269781 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.269148 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.370120 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.370045 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.370120 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.370091 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.370418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.370208 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkgkd\" (UniqueName: \"kubernetes.io/projected/ba4b9eb7-fc25-451c-9e9f-aae89573754d-kube-api-access-nkgkd\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.370418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.370257 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.370418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.370302 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.370418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.370365 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.370418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.370417 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.370713 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.370474 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.370713 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.370511 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.370713 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.370572 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.370713 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.370620 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.370876 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.370725 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.370915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.370873 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.371261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.371234 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.372905 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.372878 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.373139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.373115 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.427758 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.427726 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.427959 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.427935 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkgkd\" (UniqueName: \"kubernetes.io/projected/ba4b9eb7-fc25-451c-9e9f-aae89573754d-kube-api-access-nkgkd\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.434676 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.434643 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:27.455548 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.455517 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:27.873084 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:33:27.873054 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4b9eb7_fc25_451c_9e9f_aae89573754d.slice/crio-de3cc175f341c60e1682f5c93945d2bbc9ff00c34ac5094ca92bcee6b4016d29 WatchSource:0}: Error finding container de3cc175f341c60e1682f5c93945d2bbc9ff00c34ac5094ca92bcee6b4016d29: Status 404 returned error can't find the container with id de3cc175f341c60e1682f5c93945d2bbc9ff00c34ac5094ca92bcee6b4016d29 Nov 03 02:33:27.875940 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.875904 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:27.876048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.875975 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:27.876048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.876007 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:27.877943 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.877911 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj"] Nov 03 02:33:27.954989 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:27.954955 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" event={"ID":"ba4b9eb7-fc25-451c-9e9f-aae89573754d","Type":"ContainerStarted","Data":"de3cc175f341c60e1682f5c93945d2bbc9ff00c34ac5094ca92bcee6b4016d29"} Nov 03 02:33:28.911225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:28.911165 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:28.978463 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:28.978411 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" event={"ID":"ba4b9eb7-fc25-451c-9e9f-aae89573754d","Type":"ContainerStarted","Data":"3f52935ca2228ef3aace5345d70c6d77d4931b6d615bbfd33f74049c68ce1682"} Nov 03 02:33:29.189300 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:29.189218 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:29.228172 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:29.228122 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" podStartSLOduration=2.228107758 podStartE2EDuration="2.228107758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:29.176222448 +0000 UTC m=+2490.860331600" watchObservedRunningTime="2026-11-03 02:33:29.228107758 +0000 UTC m=+2490.912216913" Nov 03 02:33:29.357214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:29.355669 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:29.435075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:29.435045 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:29.436926 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:29.436900 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Nov 03 02:33:29.437059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:29.436951 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" podUID="ba4b9eb7-fc25-451c-9e9f-aae89573754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Nov 03 02:33:29.614011 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:29.613894 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:29.807212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:29.806933 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:30.035945 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:30.035918 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:30.451181 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:30.451133 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Nov 03 02:33:30.451490 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:30.451458 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" podUID="ba4b9eb7-fc25-451c-9e9f-aae89573754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Nov 03 02:33:31.436695 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:31.436660 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Nov 03 02:33:31.437305 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:31.437264 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" podUID="ba4b9eb7-fc25-451c-9e9f-aae89573754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Nov 03 02:33:32.436045 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:32.436007 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Nov 03 02:33:32.436266 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:32.436072 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" podUID="ba4b9eb7-fc25-451c-9e9f-aae89573754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Nov 03 02:33:32.836512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:32.836412 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:32.988482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:32.988448 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:33.133685 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:33.133594 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:33.447411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:33.447374 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Nov 03 02:33:33.453216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:33.449517 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" podUID="ba4b9eb7-fc25-451c-9e9f-aae89573754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Nov 03 02:33:34.435726 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:34.435696 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Nov 03 02:33:34.436237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:34.435750 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" podUID="ba4b9eb7-fc25-451c-9e9f-aae89573754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Nov 03 02:33:35.351515 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:35.351474 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:35.540181 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:35.540123 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:35.756673 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:35.756635 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw"] Nov 03 02:33:35.763340 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:35.763314 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw" Nov 03 02:33:35.812440 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:35.812405 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw"] Nov 03 02:33:35.870214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:35.868610 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2qnj\" (UniqueName: \"kubernetes.io/projected/edb46670-67e5-4fc2-a94d-87634bc91bc4-kube-api-access-p2qnj\") pod \"payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw\" (UID: \"edb46670-67e5-4fc2-a94d-87634bc91bc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw" Nov 03 02:33:35.870214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:35.868659 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edb46670-67e5-4fc2-a94d-87634bc91bc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw\" (UID: \"edb46670-67e5-4fc2-a94d-87634bc91bc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw" Nov 03 02:33:35.870214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:35.868970 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-f735f4\"" Nov 03 02:33:35.874687 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:35.874661 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-f735f4-dockercfg-2dgp9\"" Nov 03 02:33:35.964804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:35.964766 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb"] Nov 03 02:33:35.983519 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:35.983103 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2qnj\" (UniqueName: \"kubernetes.io/projected/edb46670-67e5-4fc2-a94d-87634bc91bc4-kube-api-access-p2qnj\") pod \"payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw\" (UID: \"edb46670-67e5-4fc2-a94d-87634bc91bc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw" Nov 03 02:33:35.983706 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:35.983626 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edb46670-67e5-4fc2-a94d-87634bc91bc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw\" (UID: \"edb46670-67e5-4fc2-a94d-87634bc91bc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw" Nov 03 02:33:35.984827 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:35.984801 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edb46670-67e5-4fc2-a94d-87634bc91bc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw\" (UID: \"edb46670-67e5-4fc2-a94d-87634bc91bc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw" Nov 03 02:33:36.006838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:36.006761 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb" Nov 03 02:33:36.011837 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:36.011809 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb"] Nov 03 02:33:36.043716 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:36.043683 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2qnj\" (UniqueName: \"kubernetes.io/projected/edb46670-67e5-4fc2-a94d-87634bc91bc4-kube-api-access-p2qnj\") pod \"payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw\" (UID: \"edb46670-67e5-4fc2-a94d-87634bc91bc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw" Nov 03 02:33:36.079935 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:36.079905 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw" Nov 03 02:33:36.084744 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:36.084722 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/699e3934-8850-4132-9f5c-3e446be27484-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb\" (UID: \"699e3934-8850-4132-9f5c-3e446be27484\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb" Nov 03 02:33:36.084876 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:36.084760 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb4cf\" (UniqueName: \"kubernetes.io/projected/699e3934-8850-4132-9f5c-3e446be27484-kube-api-access-lb4cf\") pod \"payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb\" (UID: \"699e3934-8850-4132-9f5c-3e446be27484\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb" Nov 03 02:33:36.187874 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:36.187837 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/699e3934-8850-4132-9f5c-3e446be27484-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb\" (UID: \"699e3934-8850-4132-9f5c-3e446be27484\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb" Nov 03 02:33:36.187874 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:36.187897 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb4cf\" (UniqueName: \"kubernetes.io/projected/699e3934-8850-4132-9f5c-3e446be27484-kube-api-access-lb4cf\") pod \"payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb\" (UID: \"699e3934-8850-4132-9f5c-3e446be27484\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb" Nov 03 02:33:36.189841 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:36.189146 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/699e3934-8850-4132-9f5c-3e446be27484-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb\" (UID: \"699e3934-8850-4132-9f5c-3e446be27484\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb" Nov 03 02:33:36.246514 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:36.246479 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb4cf\" (UniqueName: \"kubernetes.io/projected/699e3934-8850-4132-9f5c-3e446be27484-kube-api-access-lb4cf\") pod \"payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb\" (UID: \"699e3934-8850-4132-9f5c-3e446be27484\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb" Nov 03 02:33:36.325998 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:36.325916 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb" Nov 03 02:33:36.482021 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:36.481925 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:33:36.482021 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:36.481991 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" podUID="ba4b9eb7-fc25-451c-9e9f-aae89573754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:33:36.484274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:36.484249 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw"] Nov 03 02:33:36.718380 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:33:36.718348 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699e3934_8850_4132_9f5c_3e446be27484.slice/crio-0e4eac82bc2710c7f14c3b466e0addcc9f53858070da24b7a3563866d95441af WatchSource:0}: Error finding container 0e4eac82bc2710c7f14c3b466e0addcc9f53858070da24b7a3563866d95441af: Status 404 returned error can't find the container with id 0e4eac82bc2710c7f14c3b466e0addcc9f53858070da24b7a3563866d95441af Nov 03 02:33:36.720736 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:36.720716 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb"] Nov 03 02:33:37.039871 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:37.039790 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:37.040240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:37.040212 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:37.041482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:37.041460 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:33:37.161032 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:37.160996 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw" event={"ID":"edb46670-67e5-4fc2-a94d-87634bc91bc4","Type":"ContainerStarted","Data":"38a35ab810ef53ff3e3baa8b8486b227036a8d8b9f25c7375c0758f00abf57de"} Nov 03 02:33:37.161327 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:37.161291 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw" event={"ID":"edb46670-67e5-4fc2-a94d-87634bc91bc4","Type":"ContainerStarted","Data":"153cbd9a5474d26b6ba7badecec6155caea6bbf4467dc5895a28f5db7177c0ed"} Nov 03 02:33:37.161462 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:37.161331 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw" Nov 03 02:33:37.162906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:37.162884 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb" event={"ID":"699e3934-8850-4132-9f5c-3e446be27484","Type":"ContainerStarted","Data":"f7f40a60734173a7914d3b6f5a264ebf5dd50a16813b847d8f97ab87153f294d"} Nov 03 02:33:37.163002 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:37.162932 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb" event={"ID":"699e3934-8850-4132-9f5c-3e446be27484","Type":"ContainerStarted","Data":"0e4eac82bc2710c7f14c3b466e0addcc9f53858070da24b7a3563866d95441af"} Nov 03 02:33:37.163093 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:37.163075 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb" Nov 03 02:33:37.335472 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:37.335397 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:37.398958 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:37.398910 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb" podStartSLOduration=2.398896195 podStartE2EDuration="2.398896195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:37.396405253 +0000 UTC m=+2499.080514409" watchObservedRunningTime="2026-11-03 02:33:37.398896195 +0000 UTC m=+2499.083005383" Nov 03 02:33:37.523755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:37.523701 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw" podStartSLOduration=2.523687401 podStartE2EDuration="2.523687401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:37.521980985 +0000 UTC m=+2499.206090121" watchObservedRunningTime="2026-11-03 02:33:37.523687401 +0000 UTC m=+2499.207796556" Nov 03 02:33:37.655986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:37.655949 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj"] Nov 03 02:33:37.656213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:37.656174 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj" podUID="4c0427d1-e64c-4d1f-9169-9637e9756507" containerName="payload-processing" containerID="cri-o://7295fc25c9253d82936153419cdec49558296800589c829eaac8adebffe9b472" gracePeriod=30 Nov 03 02:33:37.708889 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:37.708864 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k"] Nov 03 02:33:37.709078 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:37.709042 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k" podUID="411eec37-7cf8-4243-b521-93c3da337d3d" containerName="payload-pre-processing" containerID="cri-o://c98513b11ffe1ab120e40997c521f4d317f6ada1f25e6ef7e34fc6f89deec00f" gracePeriod=30 Nov 03 02:33:37.834056 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:37.833753 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:37.991228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:37.988371 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:38.025233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.024208 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj" Nov 03 02:33:38.035080 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.035061 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k" Nov 03 02:33:38.128656 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.128629 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c0427d1-e64c-4d1f-9169-9637e9756507-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c0427d1-e64c-4d1f-9169-9637e9756507-plugins-config-volume\") pod \"4c0427d1-e64c-4d1f-9169-9637e9756507\" (UID: \"4c0427d1-e64c-4d1f-9169-9637e9756507\") " Nov 03 02:33:38.128858 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.128767 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c0427d1-e64c-4d1f-9169-9637e9756507-kube-api-access-xjsqq\" (UniqueName: \"kubernetes.io/projected/4c0427d1-e64c-4d1f-9169-9637e9756507-kube-api-access-xjsqq\") pod \"4c0427d1-e64c-4d1f-9169-9637e9756507\" (UID: \"4c0427d1-e64c-4d1f-9169-9637e9756507\") " Nov 03 02:33:38.129088 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.129062 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0427d1-e64c-4d1f-9169-9637e9756507-plugins-config-volume" pod "4c0427d1-e64c-4d1f-9169-9637e9756507" (UID: "4c0427d1-e64c-4d1f-9169-9637e9756507"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:33:38.129225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.129181 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c0427d1-e64c-4d1f-9169-9637e9756507-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:38.130858 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.130837 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0427d1-e64c-4d1f-9169-9637e9756507-kube-api-access-xjsqq" pod "4c0427d1-e64c-4d1f-9169-9637e9756507" (UID: "4c0427d1-e64c-4d1f-9169-9637e9756507"). InnerVolumeSpecName "kube-api-access-xjsqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:38.202269 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.189771 5463 generic.go:356] "Generic (PLEG): container finished" podID="4c0427d1-e64c-4d1f-9169-9637e9756507" containerID="7295fc25c9253d82936153419cdec49558296800589c829eaac8adebffe9b472" exitCode=0 Nov 03 02:33:38.202269 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.189895 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj" event={"ID":"4c0427d1-e64c-4d1f-9169-9637e9756507","Type":"ContainerDied","Data":"7295fc25c9253d82936153419cdec49558296800589c829eaac8adebffe9b472"} Nov 03 02:33:38.202269 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.189946 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj" event={"ID":"4c0427d1-e64c-4d1f-9169-9637e9756507","Type":"ContainerDied","Data":"9d98e2dde734150872b70694297f9ac6e7dcb5aa2b03dd6e800486864f58cbee"} Nov 03 02:33:38.202269 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.189985 5463 scope.go:122] "RemoveContainer" containerID="7295fc25c9253d82936153419cdec49558296800589c829eaac8adebffe9b472" Nov 03 02:33:38.202269 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.190180 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj" Nov 03 02:33:38.208795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.208747 5463 generic.go:356] "Generic (PLEG): container finished" podID="411eec37-7cf8-4243-b521-93c3da337d3d" containerID="c98513b11ffe1ab120e40997c521f4d317f6ada1f25e6ef7e34fc6f89deec00f" exitCode=0 Nov 03 02:33:38.209322 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.209302 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k" Nov 03 02:33:38.209764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.209654 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k" event={"ID":"411eec37-7cf8-4243-b521-93c3da337d3d","Type":"ContainerDied","Data":"c98513b11ffe1ab120e40997c521f4d317f6ada1f25e6ef7e34fc6f89deec00f"} Nov 03 02:33:38.209970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.209927 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k" event={"ID":"411eec37-7cf8-4243-b521-93c3da337d3d","Type":"ContainerDied","Data":"b4c9449ea2d721d2619b9ab1f5574e00c08be4a4e21db0cacf9c2384aa3b2503"} Nov 03 02:33:38.238342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.238317 5463 scope.go:122] "RemoveContainer" containerID="7295fc25c9253d82936153419cdec49558296800589c829eaac8adebffe9b472" Nov 03 02:33:38.238862 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:33:38.238840 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7295fc25c9253d82936153419cdec49558296800589c829eaac8adebffe9b472\": container with ID starting with 7295fc25c9253d82936153419cdec49558296800589c829eaac8adebffe9b472 not found: ID does not exist" containerID="7295fc25c9253d82936153419cdec49558296800589c829eaac8adebffe9b472" Nov 03 02:33:38.239489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.239457 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7295fc25c9253d82936153419cdec49558296800589c829eaac8adebffe9b472"} err="failed to get container status \"7295fc25c9253d82936153419cdec49558296800589c829eaac8adebffe9b472\": rpc error: code = NotFound desc = could not find container \"7295fc25c9253d82936153419cdec49558296800589c829eaac8adebffe9b472\": container with ID starting with 7295fc25c9253d82936153419cdec49558296800589c829eaac8adebffe9b472 not found: ID does not exist" Nov 03 02:33:38.239623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.239599 5463 scope.go:122] "RemoveContainer" containerID="c98513b11ffe1ab120e40997c521f4d317f6ada1f25e6ef7e34fc6f89deec00f" Nov 03 02:33:38.241639 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.241617 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/411eec37-7cf8-4243-b521-93c3da337d3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/411eec37-7cf8-4243-b521-93c3da337d3d-plugins-config-volume\") pod \"411eec37-7cf8-4243-b521-93c3da337d3d\" (UID: \"411eec37-7cf8-4243-b521-93c3da337d3d\") " Nov 03 02:33:38.241990 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.241969 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/411eec37-7cf8-4243-b521-93c3da337d3d-kube-api-access-kn9tp\" (UniqueName: \"kubernetes.io/projected/411eec37-7cf8-4243-b521-93c3da337d3d-kube-api-access-kn9tp\") pod \"411eec37-7cf8-4243-b521-93c3da337d3d\" (UID: \"411eec37-7cf8-4243-b521-93c3da337d3d\") " Nov 03 02:33:38.242908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.242887 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjsqq\" (UniqueName: \"kubernetes.io/projected/4c0427d1-e64c-4d1f-9169-9637e9756507-kube-api-access-xjsqq\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:38.243689 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.243668 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/411eec37-7cf8-4243-b521-93c3da337d3d-plugins-config-volume" pod "411eec37-7cf8-4243-b521-93c3da337d3d" (UID: "411eec37-7cf8-4243-b521-93c3da337d3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:33:38.284933 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.284432 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411eec37-7cf8-4243-b521-93c3da337d3d-kube-api-access-kn9tp" pod "411eec37-7cf8-4243-b521-93c3da337d3d" (UID: "411eec37-7cf8-4243-b521-93c3da337d3d"). InnerVolumeSpecName "kube-api-access-kn9tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:38.295755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.295433 5463 scope.go:122] "RemoveContainer" containerID="c98513b11ffe1ab120e40997c521f4d317f6ada1f25e6ef7e34fc6f89deec00f" Nov 03 02:33:38.296148 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:33:38.296043 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98513b11ffe1ab120e40997c521f4d317f6ada1f25e6ef7e34fc6f89deec00f\": container with ID starting with c98513b11ffe1ab120e40997c521f4d317f6ada1f25e6ef7e34fc6f89deec00f not found: ID does not exist" containerID="c98513b11ffe1ab120e40997c521f4d317f6ada1f25e6ef7e34fc6f89deec00f" Nov 03 02:33:38.296148 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.296083 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98513b11ffe1ab120e40997c521f4d317f6ada1f25e6ef7e34fc6f89deec00f"} err="failed to get container status \"c98513b11ffe1ab120e40997c521f4d317f6ada1f25e6ef7e34fc6f89deec00f\": rpc error: code = NotFound desc = could not find container \"c98513b11ffe1ab120e40997c521f4d317f6ada1f25e6ef7e34fc6f89deec00f\": container with ID starting with c98513b11ffe1ab120e40997c521f4d317f6ada1f25e6ef7e34fc6f89deec00f not found: ID does not exist" Nov 03 02:33:38.344866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.344826 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn9tp\" (UniqueName: \"kubernetes.io/projected/411eec37-7cf8-4243-b521-93c3da337d3d-kube-api-access-kn9tp\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:38.344866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.344866 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/411eec37-7cf8-4243-b521-93c3da337d3d-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:38.452644 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.452395 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj"] Nov 03 02:33:38.455634 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.455609 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9bff9-7ff4c99df6-fwbpj"] Nov 03 02:33:38.632667 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.632621 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k"] Nov 03 02:33:38.634780 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.634745 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9bff9-8669fbbcc5-9hl6k"] Nov 03 02:33:38.950793 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.950761 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9"] Nov 03 02:33:38.951416 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.951397 5463 cpu_manager.go:414] "containerMap: removing container" podUID="411eec37-7cf8-4243-b521-93c3da337d3d" containerName="payload-pre-processing" Nov 03 02:33:38.951511 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.951417 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="411eec37-7cf8-4243-b521-93c3da337d3d" containerName="payload-pre-processing" Nov 03 02:33:38.951511 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.951464 5463 cpu_manager.go:414] "containerMap: removing container" podUID="4c0427d1-e64c-4d1f-9169-9637e9756507" containerName="payload-processing" Nov 03 02:33:38.951511 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.951470 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c0427d1-e64c-4d1f-9169-9637e9756507" containerName="payload-processing" Nov 03 02:33:38.951629 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.951565 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c0427d1-e64c-4d1f-9169-9637e9756507" containerName="payload-processing" Nov 03 02:33:38.951629 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.951579 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="411eec37-7cf8-4243-b521-93c3da337d3d" containerName="payload-pre-processing" Nov 03 02:33:38.957044 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:38.957025 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" Nov 03 02:33:39.008068 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.008030 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9"] Nov 03 02:33:39.010337 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.010311 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:33:39.010490 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.010445 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qv59l\"" Nov 03 02:33:39.010550 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.010496 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-965a07-serving-cert\"" Nov 03 02:33:39.010596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.010572 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 02:33:39.054891 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.054864 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:33:39.055059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.055032 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f084fa9-ab75-4909-b65b-2011b6472ede-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9\" (UID: \"6f084fa9-ab75-4909-b65b-2011b6472ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" Nov 03 02:33:39.055174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.055157 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7s8f\" (UniqueName: \"kubernetes.io/projected/6f084fa9-ab75-4909-b65b-2011b6472ede-kube-api-access-b7s8f\") pod \"maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9\" (UID: \"6f084fa9-ab75-4909-b65b-2011b6472ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" Nov 03 02:33:39.055304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.055284 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f084fa9-ab75-4909-b65b-2011b6472ede-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9\" (UID: \"6f084fa9-ab75-4909-b65b-2011b6472ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" Nov 03 02:33:39.055395 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.055337 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f084fa9-ab75-4909-b65b-2011b6472ede-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9\" (UID: \"6f084fa9-ab75-4909-b65b-2011b6472ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" Nov 03 02:33:39.095409 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.095381 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:39.156809 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.156612 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f084fa9-ab75-4909-b65b-2011b6472ede-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9\" (UID: \"6f084fa9-ab75-4909-b65b-2011b6472ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" Nov 03 02:33:39.156809 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.156694 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f084fa9-ab75-4909-b65b-2011b6472ede-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9\" (UID: \"6f084fa9-ab75-4909-b65b-2011b6472ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" Nov 03 02:33:39.158245 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.156984 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f084fa9-ab75-4909-b65b-2011b6472ede-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9\" (UID: \"6f084fa9-ab75-4909-b65b-2011b6472ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" Nov 03 02:33:39.158245 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.157078 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7s8f\" (UniqueName: \"kubernetes.io/projected/6f084fa9-ab75-4909-b65b-2011b6472ede-kube-api-access-b7s8f\") pod \"maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9\" (UID: \"6f084fa9-ab75-4909-b65b-2011b6472ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" Nov 03 02:33:39.158245 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.157528 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f084fa9-ab75-4909-b65b-2011b6472ede-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9\" (UID: \"6f084fa9-ab75-4909-b65b-2011b6472ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" Nov 03 02:33:39.161458 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.161432 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f084fa9-ab75-4909-b65b-2011b6472ede-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9\" (UID: \"6f084fa9-ab75-4909-b65b-2011b6472ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" Nov 03 02:33:39.164909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.164874 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f084fa9-ab75-4909-b65b-2011b6472ede-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9\" (UID: \"6f084fa9-ab75-4909-b65b-2011b6472ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" Nov 03 02:33:39.219252 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.219127 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7s8f\" (UniqueName: \"kubernetes.io/projected/6f084fa9-ab75-4909-b65b-2011b6472ede-kube-api-access-b7s8f\") pod \"maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9\" (UID: \"6f084fa9-ab75-4909-b65b-2011b6472ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" Nov 03 02:33:39.262994 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.262965 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:39.270936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.270914 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" Nov 03 02:33:39.420529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.420501 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:39.580803 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.580713 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:39.669690 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:33:39.669656 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f084fa9_ab75_4909_b65b_2011b6472ede.slice/crio-e0ce8c36b97812260174262004334073b59b49d6fc5245be1fd93fc2f347e607 WatchSource:0}: Error finding container e0ce8c36b97812260174262004334073b59b49d6fc5245be1fd93fc2f347e607: Status 404 returned error can't find the container with id e0ce8c36b97812260174262004334073b59b49d6fc5245be1fd93fc2f347e607 Nov 03 02:33:39.671233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.671179 5463 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 02:33:39.672902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:39.672873 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9"] Nov 03 02:33:40.228959 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:40.228922 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" event={"ID":"6f084fa9-ab75-4909-b65b-2011b6472ede","Type":"ContainerStarted","Data":"e0ce8c36b97812260174262004334073b59b49d6fc5245be1fd93fc2f347e607"} Nov 03 02:33:40.443536 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:40.443504 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="411eec37-7cf8-4243-b521-93c3da337d3d" path="/var/lib/kubelet/pods/411eec37-7cf8-4243-b521-93c3da337d3d/volumes" Nov 03 02:33:40.444023 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:40.444002 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c0427d1-e64c-4d1f-9169-9637e9756507" path="/var/lib/kubelet/pods/4c0427d1-e64c-4d1f-9169-9637e9756507/volumes" Nov 03 02:33:41.037703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:41.037662 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:41.238992 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:41.238955 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" event={"ID":"6f084fa9-ab75-4909-b65b-2011b6472ede","Type":"ContainerStarted","Data":"984ed74a227e3fb90e649ac3ec1013707d912c9861112785d041a4e22b749cdf"} Nov 03 02:33:41.239417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:41.239170 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" Nov 03 02:33:41.357156 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:41.357060 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" podStartSLOduration=2.451878519 podStartE2EDuration="3.357046307s" totalImagesPullingTime="905.167788ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:41.355464946 +0000 UTC m=+2503.039574113" watchObservedRunningTime="2026-11-03 02:33:41.357046307 +0000 UTC m=+2503.041155497" Nov 03 02:33:41.485269 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:41.484454 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl"] Nov 03 02:33:41.485269 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:41.484706 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" podUID="da4fd375-fcd1-4665-8a2e-89a67bb9dc43" containerName="istio-proxy" containerID="cri-o://9d4857cba5b69b46689d1c572dc201c474aaa59e3a5a1c33c9755bc5ceb5f7c1" gracePeriod=30 Nov 03 02:33:41.497923 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:41.497888 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz"] Nov 03 02:33:41.498162 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:41.498136 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" podUID="21beedd4-fac4-4b92-9cf8-c2acb85422f4" containerName="istio-proxy" containerID="cri-o://ec2f75958173a8cfbb9f5920d5000fa0da79d56ef2b1f8be4d68a9c7b8b08a48" gracePeriod=30 Nov 03 02:33:41.499349 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:41.499316 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2"] Nov 03 02:33:41.499550 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:41.499520 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" podUID="ad2720fd-aa19-4213-a377-c98474134c93" containerName="istio-proxy" containerID="cri-o://abdd6febda5525eb728fb19fa6869d93e56f230bfbb19cad71001345aed21110" gracePeriod=30 Nov 03 02:33:41.511173 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:41.511121 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4"] Nov 03 02:33:41.511436 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:41.511375 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" podUID="e1bab284-6e6d-4ace-a4af-65fd25483815" containerName="istio-proxy" containerID="cri-o://3556c9e31dd5ed5f894b45a5d03560a1792546d9015fbdfc2d236e66fa274c22" gracePeriod=30 Nov 03 02:33:41.528207 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:41.528157 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:42.212450 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.212416 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf"] Nov 03 02:33:42.216888 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.216869 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.273090 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.273062 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf"] Nov 03 02:33:42.303312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.303282 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.303507 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.303343 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.303507 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.303373 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.303507 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.303432 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.304148 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.303525 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.304148 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.303553 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.304148 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.303624 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.304148 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.303663 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.304148 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.303723 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg89m\" (UniqueName: \"kubernetes.io/projected/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-kube-api-access-wg89m\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.404841 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.404803 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.405033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.404852 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.405033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.404868 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.405033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.404903 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.405033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.405007 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.405033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.405031 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.405322 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.405129 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.405322 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.405247 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.405429 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.405330 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg89m\" (UniqueName: \"kubernetes.io/projected/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-kube-api-access-wg89m\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.405429 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.405401 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.405536 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.405475 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.405590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.405550 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.405838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.405815 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.405923 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.405906 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.407285 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.407259 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.407607 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.407585 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.462056 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.462025 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg89m\" (UniqueName: \"kubernetes.io/projected/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-kube-api-access-wg89m\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.462410 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.462387 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.530727 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.530662 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:42.964241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.964170 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf"] Nov 03 02:33:42.964528 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:33:42.964500 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9db59ca_ab6b_4c1d_b24c_bcea3efc4268.slice/crio-7b8b3a16316e4eb1c76dddd245bab36eb361610689ff003daf34bac066bfac75 WatchSource:0}: Error finding container 7b8b3a16316e4eb1c76dddd245bab36eb361610689ff003daf34bac066bfac75: Status 404 returned error can't find the container with id 7b8b3a16316e4eb1c76dddd245bab36eb361610689ff003daf34bac066bfac75 Nov 03 02:33:42.967382 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.967342 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:42.967477 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.967424 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:42.967477 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:42.967464 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:43.012576 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:33:43.012541 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:33:43.108728 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:43.108699 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:43.259829 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:43.259743 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" event={"ID":"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268","Type":"ContainerStarted","Data":"0354686078f514d804a4f824641bf6f01d75020b6bf11b89a176ddc768bbcf35"} Nov 03 02:33:43.259829 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:43.259816 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" event={"ID":"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268","Type":"ContainerStarted","Data":"7b8b3a16316e4eb1c76dddd245bab36eb361610689ff003daf34bac066bfac75"} Nov 03 02:33:43.384783 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:43.384732 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" podStartSLOduration=1.384716169 podStartE2EDuration="1.384716169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:43.382485557 +0000 UTC m=+2505.066594712" watchObservedRunningTime="2026-11-03 02:33:43.384716169 +0000 UTC m=+2505.068825325" Nov 03 02:33:43.531512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:43.531438 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:43.603497 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:43.603464 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh"] Nov 03 02:33:43.607999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:43.607980 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:43.702801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:43.675104 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-f954db-openshift-default-dockercfg-nb7bd\"" Nov 03 02:33:43.786267 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:43.780685 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:43.801215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:43.788154 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh"] Nov 03 02:33:43.914785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:43.914050 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-podinfo\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:43.914785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:43.914097 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-envoy\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:43.914785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:43.914146 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-workload-certs\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:43.914785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:43.914218 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-data\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:43.914785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:43.914278 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-token\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:43.914785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:43.914337 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-workload-socket\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:43.914785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:43.914387 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpz4p\" (UniqueName: \"kubernetes.io/projected/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-kube-api-access-zpz4p\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:43.914785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:43.914514 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-credential-socket\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:43.914785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:43.914549 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istiod-ca-cert\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:44.015907 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.015872 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-credential-socket\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:44.015907 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.015912 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istiod-ca-cert\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:44.016225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.015974 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-podinfo\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:44.016225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.015995 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-envoy\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:44.016225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.016030 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-workload-certs\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:44.016225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.016075 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-data\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:44.016225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.016127 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-token\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:44.016225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.016179 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-workload-socket\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:44.016591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.016249 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpz4p\" (UniqueName: \"kubernetes.io/projected/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-kube-api-access-zpz4p\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:44.016591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.016396 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-credential-socket\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:44.016591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.016436 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-workload-certs\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:44.016998 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.016641 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-workload-socket\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:44.016998 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.016675 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-data\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:44.016998 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.016797 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istiod-ca-cert\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:44.018574 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.018555 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-envoy\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:44.018803 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.018783 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-podinfo\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:44.084863 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.084750 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-token\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:44.086084 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.086032 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpz4p\" (UniqueName: \"kubernetes.io/projected/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-kube-api-access-zpz4p\") pod \"e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:44.162217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.154076 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:44.230678 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.230648 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:44.532384 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.532352 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 03 02:33:44.532772 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.532401 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" podUID="a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 03 02:33:44.636529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.636499 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:44.702959 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.702922 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh"] Nov 03 02:33:44.718178 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:33:44.718146 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec0591b_d4b8_43f5_a031_6ee4e22c96e5.slice/crio-160964fddcb04a8cd6e2c81ade46d9856fa055eaacca5df3164780f61058d5da WatchSource:0}: Error finding container 160964fddcb04a8cd6e2c81ade46d9856fa055eaacca5df3164780f61058d5da: Status 404 returned error can't find the container with id 160964fddcb04a8cd6e2c81ade46d9856fa055eaacca5df3164780f61058d5da Nov 03 02:33:44.721014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.720971 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:44.721110 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.721066 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:44.721173 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:44.721109 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:33:45.302305 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:45.302264 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" event={"ID":"cec0591b-d4b8-43f5-a031-6ee4e22c96e5","Type":"ContainerStarted","Data":"9597e9b503e48921142c030e540a31efcba4a883220ceb8ee61a4cb3f24081a3"} Nov 03 02:33:45.302490 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:45.302335 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" event={"ID":"cec0591b-d4b8-43f5-a031-6ee4e22c96e5","Type":"ContainerStarted","Data":"160964fddcb04a8cd6e2c81ade46d9856fa055eaacca5df3164780f61058d5da"} Nov 03 02:33:45.480147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:45.480069 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" podStartSLOduration=2.480046633 podStartE2EDuration="2.480046633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:45.44711388 +0000 UTC m=+2507.131223036" watchObservedRunningTime="2026-11-03 02:33:45.480046633 +0000 UTC m=+2507.164155789" Nov 03 02:33:45.530999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:45.530969 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 03 02:33:45.531139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:45.531018 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" podUID="a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 03 02:33:46.231567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:46.231535 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:46.237071 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:46.237044 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:46.311123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:46.311093 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:46.312166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:46.312144 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:33:46.531345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:46.531318 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 03 02:33:46.531476 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:46.531369 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" podUID="a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 03 02:33:46.841228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:46.841185 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:33:46.944326 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:46.944302 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:33:47.005166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.004223 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-credential-socket\") pod \"e1bab284-6e6d-4ace-a4af-65fd25483815\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " Nov 03 02:33:47.005166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.004304 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-podinfo\") pod \"e1bab284-6e6d-4ace-a4af-65fd25483815\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " Nov 03 02:33:47.005166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.004325 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-workload-socket\") pod \"e1bab284-6e6d-4ace-a4af-65fd25483815\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " Nov 03 02:33:47.005166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.004370 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-workload-certs\") pod \"e1bab284-6e6d-4ace-a4af-65fd25483815\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " Nov 03 02:33:47.005166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.004424 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1bab284-6e6d-4ace-a4af-65fd25483815-kube-api-access-jklgb\" (UniqueName: \"kubernetes.io/projected/e1bab284-6e6d-4ace-a4af-65fd25483815-kube-api-access-jklgb\") pod \"e1bab284-6e6d-4ace-a4af-65fd25483815\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " Nov 03 02:33:47.005166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.004485 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-data\") pod \"e1bab284-6e6d-4ace-a4af-65fd25483815\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " Nov 03 02:33:47.005166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.004518 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-envoy\") pod \"e1bab284-6e6d-4ace-a4af-65fd25483815\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " Nov 03 02:33:47.005166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.004542 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-token\" (UniqueName: \"kubernetes.io/projected/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-token\") pod \"e1bab284-6e6d-4ace-a4af-65fd25483815\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " Nov 03 02:33:47.005166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.004607 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1bab284-6e6d-4ace-a4af-65fd25483815-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1bab284-6e6d-4ace-a4af-65fd25483815-istiod-ca-cert\") pod \"e1bab284-6e6d-4ace-a4af-65fd25483815\" (UID: \"e1bab284-6e6d-4ace-a4af-65fd25483815\") " Nov 03 02:33:47.005896 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.005771 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1bab284-6e6d-4ace-a4af-65fd25483815-istiod-ca-cert" pod "e1bab284-6e6d-4ace-a4af-65fd25483815" (UID: "e1bab284-6e6d-4ace-a4af-65fd25483815"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:33:47.006814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.006756 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-workload-certs" pod "e1bab284-6e6d-4ace-a4af-65fd25483815" (UID: "e1bab284-6e6d-4ace-a4af-65fd25483815"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:47.007012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.006965 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-credential-socket" pod "e1bab284-6e6d-4ace-a4af-65fd25483815" (UID: "e1bab284-6e6d-4ace-a4af-65fd25483815"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:47.008529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.008244 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-workload-socket" pod "e1bab284-6e6d-4ace-a4af-65fd25483815" (UID: "e1bab284-6e6d-4ace-a4af-65fd25483815"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:47.009300 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.009278 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-data" pod "e1bab284-6e6d-4ace-a4af-65fd25483815" (UID: "e1bab284-6e6d-4ace-a4af-65fd25483815"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:47.013091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.013069 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-token" pod "e1bab284-6e6d-4ace-a4af-65fd25483815" (UID: "e1bab284-6e6d-4ace-a4af-65fd25483815"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:47.013726 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.013677 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1bab284-6e6d-4ace-a4af-65fd25483815-kube-api-access-jklgb" pod "e1bab284-6e6d-4ace-a4af-65fd25483815" (UID: "e1bab284-6e6d-4ace-a4af-65fd25483815"). InnerVolumeSpecName "kube-api-access-jklgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:47.015317 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.015295 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-podinfo" pod "e1bab284-6e6d-4ace-a4af-65fd25483815" (UID: "e1bab284-6e6d-4ace-a4af-65fd25483815"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:33:47.018059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.018038 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-envoy" pod "e1bab284-6e6d-4ace-a4af-65fd25483815" (UID: "e1bab284-6e6d-4ace-a4af-65fd25483815"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:47.022420 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.020318 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:33:47.037118 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.037081 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:33:47.106118 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.106027 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istiod-ca-cert\") pod \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " Nov 03 02:33:47.106118 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.106088 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-workload-certs\") pod \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " Nov 03 02:33:47.106418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.106119 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-envoy\") pod \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " Nov 03 02:33:47.106418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.106153 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-token\" (UniqueName: \"kubernetes.io/projected/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-token\") pod \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " Nov 03 02:33:47.106418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.106203 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-podinfo\") pod \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " Nov 03 02:33:47.106418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.106234 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-token\" (UniqueName: \"kubernetes.io/projected/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-token\") pod \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " Nov 03 02:33:47.106418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.106294 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-envoy\") pod \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " Nov 03 02:33:47.106418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.106391 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21beedd4-fac4-4b92-9cf8-c2acb85422f4-kube-api-access-xkgvj\" (UniqueName: \"kubernetes.io/projected/21beedd4-fac4-4b92-9cf8-c2acb85422f4-kube-api-access-xkgvj\") pod \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " Nov 03 02:33:47.106765 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.106463 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-podinfo\") pod \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " Nov 03 02:33:47.106765 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.106492 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-kube-api-access-fdccg\" (UniqueName: \"kubernetes.io/projected/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-kube-api-access-fdccg\") pod \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " Nov 03 02:33:47.106765 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.106561 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-workload-socket\") pod \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " Nov 03 02:33:47.106765 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.106613 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-workload-socket\") pod \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " Nov 03 02:33:47.106765 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.106662 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-credential-socket\") pod \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " Nov 03 02:33:47.106765 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.106699 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istiod-ca-cert\") pod \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " Nov 03 02:33:47.107101 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.106770 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-data\") pod \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\" (UID: \"da4fd375-fcd1-4665-8a2e-89a67bb9dc43\") " Nov 03 02:33:47.107101 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.106800 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-data\") pod \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " Nov 03 02:33:47.107101 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.106871 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-credential-socket\") pod \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " Nov 03 02:33:47.107101 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.106920 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-workload-certs\") pod \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\" (UID: \"21beedd4-fac4-4b92-9cf8-c2acb85422f4\") " Nov 03 02:33:47.107345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.107298 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-workload-certs" pod "da4fd375-fcd1-4665-8a2e-89a67bb9dc43" (UID: "da4fd375-fcd1-4665-8a2e-89a67bb9dc43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:47.107528 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.107506 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-workload-certs" pod "21beedd4-fac4-4b92-9cf8-c2acb85422f4" (UID: "21beedd4-fac4-4b92-9cf8-c2acb85422f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:47.109955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.109702 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-workload-socket" pod "21beedd4-fac4-4b92-9cf8-c2acb85422f4" (UID: "21beedd4-fac4-4b92-9cf8-c2acb85422f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:47.110096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.110061 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-envoy" pod "21beedd4-fac4-4b92-9cf8-c2acb85422f4" (UID: "21beedd4-fac4-4b92-9cf8-c2acb85422f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:47.110576 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.109884 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-workload-socket" pod "da4fd375-fcd1-4665-8a2e-89a67bb9dc43" (UID: "da4fd375-fcd1-4665-8a2e-89a67bb9dc43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:47.110694 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.110627 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istiod-ca-cert" pod "da4fd375-fcd1-4665-8a2e-89a67bb9dc43" (UID: "da4fd375-fcd1-4665-8a2e-89a67bb9dc43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:33:47.110768 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.110751 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-credential-socket" pod "da4fd375-fcd1-4665-8a2e-89a67bb9dc43" (UID: "da4fd375-fcd1-4665-8a2e-89a67bb9dc43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:47.111302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.110931 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1bab284-6e6d-4ace-a4af-65fd25483815-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.111302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.110958 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.111302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.110974 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.111302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.110988 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.111302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.111003 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.111302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.111005 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istiod-ca-cert" pod "21beedd4-fac4-4b92-9cf8-c2acb85422f4" (UID: "21beedd4-fac4-4b92-9cf8-c2acb85422f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:33:47.111302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.111017 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.111302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.111033 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.111302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.111090 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.111302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.111104 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.111302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.111121 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jklgb\" (UniqueName: \"kubernetes.io/projected/e1bab284-6e6d-4ace-a4af-65fd25483815-kube-api-access-jklgb\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.111302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.111135 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.111302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.111151 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.111302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.111228 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.111302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.111243 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.111302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.111257 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.111302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.111274 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1bab284-6e6d-4ace-a4af-65fd25483815-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.112423 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.111257 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-data" pod "da4fd375-fcd1-4665-8a2e-89a67bb9dc43" (UID: "da4fd375-fcd1-4665-8a2e-89a67bb9dc43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:47.112423 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.111445 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-data" pod "21beedd4-fac4-4b92-9cf8-c2acb85422f4" (UID: "21beedd4-fac4-4b92-9cf8-c2acb85422f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:47.112423 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.111620 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-credential-socket" pod "21beedd4-fac4-4b92-9cf8-c2acb85422f4" (UID: "21beedd4-fac4-4b92-9cf8-c2acb85422f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:47.112597 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.112566 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-kube-api-access-fdccg" pod "da4fd375-fcd1-4665-8a2e-89a67bb9dc43" (UID: "da4fd375-fcd1-4665-8a2e-89a67bb9dc43"). InnerVolumeSpecName "kube-api-access-fdccg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:47.113839 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.113803 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-podinfo" pod "da4fd375-fcd1-4665-8a2e-89a67bb9dc43" (UID: "da4fd375-fcd1-4665-8a2e-89a67bb9dc43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:33:47.114096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.114071 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-token" pod "21beedd4-fac4-4b92-9cf8-c2acb85422f4" (UID: "21beedd4-fac4-4b92-9cf8-c2acb85422f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:47.115688 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.114701 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-envoy" pod "da4fd375-fcd1-4665-8a2e-89a67bb9dc43" (UID: "da4fd375-fcd1-4665-8a2e-89a67bb9dc43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:47.115688 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.115071 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-token" pod "da4fd375-fcd1-4665-8a2e-89a67bb9dc43" (UID: "da4fd375-fcd1-4665-8a2e-89a67bb9dc43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:47.117019 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.116984 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21beedd4-fac4-4b92-9cf8-c2acb85422f4-kube-api-access-xkgvj" pod "21beedd4-fac4-4b92-9cf8-c2acb85422f4" (UID: "21beedd4-fac4-4b92-9cf8-c2acb85422f4"). InnerVolumeSpecName "kube-api-access-xkgvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:47.117165 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.117142 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-podinfo" pod "21beedd4-fac4-4b92-9cf8-c2acb85422f4" (UID: "21beedd4-fac4-4b92-9cf8-c2acb85422f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:33:47.216614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.216580 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad2720fd-aa19-4213-a377-c98474134c93-istio-token\" (UniqueName: \"kubernetes.io/projected/ad2720fd-aa19-4213-a377-c98474134c93-istio-token\") pod \"ad2720fd-aa19-4213-a377-c98474134c93\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " Nov 03 02:33:47.216814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.216645 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-workload-certs\") pod \"ad2720fd-aa19-4213-a377-c98474134c93\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " Nov 03 02:33:47.216814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.216700 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad2720fd-aa19-4213-a377-c98474134c93-kube-api-access-kvz9q\" (UniqueName: \"kubernetes.io/projected/ad2720fd-aa19-4213-a377-c98474134c93-kube-api-access-kvz9q\") pod \"ad2720fd-aa19-4213-a377-c98474134c93\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " Nov 03 02:33:47.216814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.216763 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad2720fd-aa19-4213-a377-c98474134c93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad2720fd-aa19-4213-a377-c98474134c93-istio-podinfo\") pod \"ad2720fd-aa19-4213-a377-c98474134c93\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " Nov 03 02:33:47.216814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.216796 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-istio-data\") pod \"ad2720fd-aa19-4213-a377-c98474134c93\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " Nov 03 02:33:47.217057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.216923 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad2720fd-aa19-4213-a377-c98474134c93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad2720fd-aa19-4213-a377-c98474134c93-istiod-ca-cert\") pod \"ad2720fd-aa19-4213-a377-c98474134c93\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " Nov 03 02:33:47.217057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.216976 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-workload-socket\") pod \"ad2720fd-aa19-4213-a377-c98474134c93\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " Nov 03 02:33:47.217204 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.217091 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-credential-socket\") pod \"ad2720fd-aa19-4213-a377-c98474134c93\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " Nov 03 02:33:47.217204 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.217163 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-istio-envoy\") pod \"ad2720fd-aa19-4213-a377-c98474134c93\" (UID: \"ad2720fd-aa19-4213-a377-c98474134c93\") " Nov 03 02:33:47.217769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.217744 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.217884 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.217772 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdccg\" (UniqueName: \"kubernetes.io/projected/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-kube-api-access-fdccg\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.217884 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.217793 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.217884 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.217808 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.217884 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.217821 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.217884 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.217836 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21beedd4-fac4-4b92-9cf8-c2acb85422f4-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.217884 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.217849 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.217884 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.217862 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21beedd4-fac4-4b92-9cf8-c2acb85422f4-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.217884 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.217876 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.218365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.217891 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4fd375-fcd1-4665-8a2e-89a67bb9dc43-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.218365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.217905 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkgvj\" (UniqueName: \"kubernetes.io/projected/21beedd4-fac4-4b92-9cf8-c2acb85422f4-kube-api-access-xkgvj\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.218365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.217939 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-workload-socket" pod "ad2720fd-aa19-4213-a377-c98474134c93" (UID: "ad2720fd-aa19-4213-a377-c98474134c93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:47.218365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.218168 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-workload-certs" pod "ad2720fd-aa19-4213-a377-c98474134c93" (UID: "ad2720fd-aa19-4213-a377-c98474134c93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:47.218585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.218562 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2720fd-aa19-4213-a377-c98474134c93-istiod-ca-cert" pod "ad2720fd-aa19-4213-a377-c98474134c93" (UID: "ad2720fd-aa19-4213-a377-c98474134c93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:33:47.218651 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.218634 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-credential-socket" pod "ad2720fd-aa19-4213-a377-c98474134c93" (UID: "ad2720fd-aa19-4213-a377-c98474134c93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:47.218919 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.218894 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-istio-data" pod "ad2720fd-aa19-4213-a377-c98474134c93" (UID: "ad2720fd-aa19-4213-a377-c98474134c93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:47.221941 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.221901 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2720fd-aa19-4213-a377-c98474134c93-istio-token" pod "ad2720fd-aa19-4213-a377-c98474134c93" (UID: "ad2720fd-aa19-4213-a377-c98474134c93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:47.223049 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.223021 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad2720fd-aa19-4213-a377-c98474134c93-istio-podinfo" pod "ad2720fd-aa19-4213-a377-c98474134c93" (UID: "ad2720fd-aa19-4213-a377-c98474134c93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:33:47.223909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.223888 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-istio-envoy" pod "ad2720fd-aa19-4213-a377-c98474134c93" (UID: "ad2720fd-aa19-4213-a377-c98474134c93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:33:47.226465 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.226443 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2720fd-aa19-4213-a377-c98474134c93-kube-api-access-kvz9q" pod "ad2720fd-aa19-4213-a377-c98474134c93" (UID: "ad2720fd-aa19-4213-a377-c98474134c93"). InnerVolumeSpecName "kube-api-access-kvz9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:47.284182 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.284152 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" Nov 03 02:33:47.319267 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.319230 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad2720fd-aa19-4213-a377-c98474134c93-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.319267 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.319266 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.319473 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.319281 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad2720fd-aa19-4213-a377-c98474134c93-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.319473 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.319295 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.319473 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.319309 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.319473 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.319325 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.319473 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.319339 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad2720fd-aa19-4213-a377-c98474134c93-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.319473 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.319353 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad2720fd-aa19-4213-a377-c98474134c93-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.319473 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.319367 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvz9q\" (UniqueName: \"kubernetes.io/projected/ad2720fd-aa19-4213-a377-c98474134c93-kube-api-access-kvz9q\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:47.321153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.321123 5463 generic.go:356] "Generic (PLEG): container finished" podID="21beedd4-fac4-4b92-9cf8-c2acb85422f4" containerID="ec2f75958173a8cfbb9f5920d5000fa0da79d56ef2b1f8be4d68a9c7b8b08a48" exitCode=0 Nov 03 02:33:47.321295 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.321170 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" Nov 03 02:33:47.321295 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.321220 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" event={"ID":"21beedd4-fac4-4b92-9cf8-c2acb85422f4","Type":"ContainerDied","Data":"ec2f75958173a8cfbb9f5920d5000fa0da79d56ef2b1f8be4d68a9c7b8b08a48"} Nov 03 02:33:47.321295 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.321273 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz" event={"ID":"21beedd4-fac4-4b92-9cf8-c2acb85422f4","Type":"ContainerDied","Data":"b92bbaad9462c7ac79c520891381f7fdb4d5ba7f077f69b89599d94f688fd8bb"} Nov 03 02:33:47.321453 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.321300 5463 scope.go:122] "RemoveContainer" containerID="ec2f75958173a8cfbb9f5920d5000fa0da79d56ef2b1f8be4d68a9c7b8b08a48" Nov 03 02:33:47.322931 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.322908 5463 generic.go:356] "Generic (PLEG): container finished" podID="e1bab284-6e6d-4ace-a4af-65fd25483815" containerID="3556c9e31dd5ed5f894b45a5d03560a1792546d9015fbdfc2d236e66fa274c22" exitCode=0 Nov 03 02:33:47.323057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.322974 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" Nov 03 02:33:47.323057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.322994 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" event={"ID":"e1bab284-6e6d-4ace-a4af-65fd25483815","Type":"ContainerDied","Data":"3556c9e31dd5ed5f894b45a5d03560a1792546d9015fbdfc2d236e66fa274c22"} Nov 03 02:33:47.323057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.323038 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4" event={"ID":"e1bab284-6e6d-4ace-a4af-65fd25483815","Type":"ContainerDied","Data":"bd41caa59ef750ecf1ea324857f18d5bc5e60f0f15c15fa18267ce51b2b2e458"} Nov 03 02:33:47.329680 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.329660 5463 generic.go:356] "Generic (PLEG): container finished" podID="da4fd375-fcd1-4665-8a2e-89a67bb9dc43" containerID="9d4857cba5b69b46689d1c572dc201c474aaa59e3a5a1c33c9755bc5ceb5f7c1" exitCode=0 Nov 03 02:33:47.329823 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.329717 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" Nov 03 02:33:47.329823 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.329744 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" event={"ID":"da4fd375-fcd1-4665-8a2e-89a67bb9dc43","Type":"ContainerDied","Data":"9d4857cba5b69b46689d1c572dc201c474aaa59e3a5a1c33c9755bc5ceb5f7c1"} Nov 03 02:33:47.329823 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.329790 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl" event={"ID":"da4fd375-fcd1-4665-8a2e-89a67bb9dc43","Type":"ContainerDied","Data":"c9faedd5fb11c70f578e27684718889e0ccf3c1c895f58af5e3d02588ece6ed7"} Nov 03 02:33:47.332234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.332121 5463 generic.go:356] "Generic (PLEG): container finished" podID="ad2720fd-aa19-4213-a377-c98474134c93" containerID="abdd6febda5525eb728fb19fa6869d93e56f230bfbb19cad71001345aed21110" exitCode=0 Nov 03 02:33:47.332234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.332151 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" event={"ID":"ad2720fd-aa19-4213-a377-c98474134c93","Type":"ContainerDied","Data":"abdd6febda5525eb728fb19fa6869d93e56f230bfbb19cad71001345aed21110"} Nov 03 02:33:47.332234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.332224 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" event={"ID":"ad2720fd-aa19-4213-a377-c98474134c93","Type":"ContainerDied","Data":"13637b00c35bf69e3292f37d2bd1797f4ef1f89630945de7688be9d3f0634e43"} Nov 03 02:33:47.332497 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.332181 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2" Nov 03 02:33:47.338658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.338642 5463 scope.go:122] "RemoveContainer" containerID="ec2f75958173a8cfbb9f5920d5000fa0da79d56ef2b1f8be4d68a9c7b8b08a48" Nov 03 02:33:47.345264 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:33:47.341044 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec2f75958173a8cfbb9f5920d5000fa0da79d56ef2b1f8be4d68a9c7b8b08a48\": container with ID starting with ec2f75958173a8cfbb9f5920d5000fa0da79d56ef2b1f8be4d68a9c7b8b08a48 not found: ID does not exist" containerID="ec2f75958173a8cfbb9f5920d5000fa0da79d56ef2b1f8be4d68a9c7b8b08a48" Nov 03 02:33:47.345264 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.341078 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec2f75958173a8cfbb9f5920d5000fa0da79d56ef2b1f8be4d68a9c7b8b08a48"} err="failed to get container status \"ec2f75958173a8cfbb9f5920d5000fa0da79d56ef2b1f8be4d68a9c7b8b08a48\": rpc error: code = NotFound desc = could not find container \"ec2f75958173a8cfbb9f5920d5000fa0da79d56ef2b1f8be4d68a9c7b8b08a48\": container with ID starting with ec2f75958173a8cfbb9f5920d5000fa0da79d56ef2b1f8be4d68a9c7b8b08a48 not found: ID does not exist" Nov 03 02:33:47.345264 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.341095 5463 scope.go:122] "RemoveContainer" containerID="3556c9e31dd5ed5f894b45a5d03560a1792546d9015fbdfc2d236e66fa274c22" Nov 03 02:33:47.392787 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.392763 5463 scope.go:122] "RemoveContainer" containerID="3556c9e31dd5ed5f894b45a5d03560a1792546d9015fbdfc2d236e66fa274c22" Nov 03 02:33:47.393345 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:33:47.393267 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3556c9e31dd5ed5f894b45a5d03560a1792546d9015fbdfc2d236e66fa274c22\": container with ID starting with 3556c9e31dd5ed5f894b45a5d03560a1792546d9015fbdfc2d236e66fa274c22 not found: ID does not exist" containerID="3556c9e31dd5ed5f894b45a5d03560a1792546d9015fbdfc2d236e66fa274c22" Nov 03 02:33:47.393452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.393362 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3556c9e31dd5ed5f894b45a5d03560a1792546d9015fbdfc2d236e66fa274c22"} err="failed to get container status \"3556c9e31dd5ed5f894b45a5d03560a1792546d9015fbdfc2d236e66fa274c22\": rpc error: code = NotFound desc = could not find container \"3556c9e31dd5ed5f894b45a5d03560a1792546d9015fbdfc2d236e66fa274c22\": container with ID starting with 3556c9e31dd5ed5f894b45a5d03560a1792546d9015fbdfc2d236e66fa274c22 not found: ID does not exist" Nov 03 02:33:47.393452 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.393389 5463 scope.go:122] "RemoveContainer" containerID="9d4857cba5b69b46689d1c572dc201c474aaa59e3a5a1c33c9755bc5ceb5f7c1" Nov 03 02:33:47.404915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.404889 5463 scope.go:122] "RemoveContainer" containerID="4b6e748def28d2bc8156b0730cc50284d393f153af9ed4bee58ea6d6050e1dbf" Nov 03 02:33:47.414777 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.414746 5463 scope.go:122] "RemoveContainer" containerID="9d4857cba5b69b46689d1c572dc201c474aaa59e3a5a1c33c9755bc5ceb5f7c1" Nov 03 02:33:47.415046 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:33:47.415026 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d4857cba5b69b46689d1c572dc201c474aaa59e3a5a1c33c9755bc5ceb5f7c1\": container with ID starting with 9d4857cba5b69b46689d1c572dc201c474aaa59e3a5a1c33c9755bc5ceb5f7c1 not found: ID does not exist" containerID="9d4857cba5b69b46689d1c572dc201c474aaa59e3a5a1c33c9755bc5ceb5f7c1" Nov 03 02:33:47.415121 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.415059 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4857cba5b69b46689d1c572dc201c474aaa59e3a5a1c33c9755bc5ceb5f7c1"} err="failed to get container status \"9d4857cba5b69b46689d1c572dc201c474aaa59e3a5a1c33c9755bc5ceb5f7c1\": rpc error: code = NotFound desc = could not find container \"9d4857cba5b69b46689d1c572dc201c474aaa59e3a5a1c33c9755bc5ceb5f7c1\": container with ID starting with 9d4857cba5b69b46689d1c572dc201c474aaa59e3a5a1c33c9755bc5ceb5f7c1 not found: ID does not exist" Nov 03 02:33:47.415121 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.415077 5463 scope.go:122] "RemoveContainer" containerID="4b6e748def28d2bc8156b0730cc50284d393f153af9ed4bee58ea6d6050e1dbf" Nov 03 02:33:47.415347 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:33:47.415329 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b6e748def28d2bc8156b0730cc50284d393f153af9ed4bee58ea6d6050e1dbf\": container with ID starting with 4b6e748def28d2bc8156b0730cc50284d393f153af9ed4bee58ea6d6050e1dbf not found: ID does not exist" containerID="4b6e748def28d2bc8156b0730cc50284d393f153af9ed4bee58ea6d6050e1dbf" Nov 03 02:33:47.415406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.415355 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b6e748def28d2bc8156b0730cc50284d393f153af9ed4bee58ea6d6050e1dbf"} err="failed to get container status \"4b6e748def28d2bc8156b0730cc50284d393f153af9ed4bee58ea6d6050e1dbf\": rpc error: code = NotFound desc = could not find container \"4b6e748def28d2bc8156b0730cc50284d393f153af9ed4bee58ea6d6050e1dbf\": container with ID starting with 4b6e748def28d2bc8156b0730cc50284d393f153af9ed4bee58ea6d6050e1dbf not found: ID does not exist" Nov 03 02:33:47.415406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.415374 5463 scope.go:122] "RemoveContainer" containerID="abdd6febda5525eb728fb19fa6869d93e56f230bfbb19cad71001345aed21110" Nov 03 02:33:47.424291 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.424273 5463 scope.go:122] "RemoveContainer" containerID="abdd6febda5525eb728fb19fa6869d93e56f230bfbb19cad71001345aed21110" Nov 03 02:33:47.424560 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:33:47.424532 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abdd6febda5525eb728fb19fa6869d93e56f230bfbb19cad71001345aed21110\": container with ID starting with abdd6febda5525eb728fb19fa6869d93e56f230bfbb19cad71001345aed21110 not found: ID does not exist" containerID="abdd6febda5525eb728fb19fa6869d93e56f230bfbb19cad71001345aed21110" Nov 03 02:33:47.424630 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.424558 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abdd6febda5525eb728fb19fa6869d93e56f230bfbb19cad71001345aed21110"} err="failed to get container status \"abdd6febda5525eb728fb19fa6869d93e56f230bfbb19cad71001345aed21110\": rpc error: code = NotFound desc = could not find container \"abdd6febda5525eb728fb19fa6869d93e56f230bfbb19cad71001345aed21110\": container with ID starting with abdd6febda5525eb728fb19fa6869d93e56f230bfbb19cad71001345aed21110 not found: ID does not exist" Nov 03 02:33:47.531883 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.531845 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 03 02:33:47.532041 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.531906 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" podUID="a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 03 02:33:47.594372 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.594310 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz"] Nov 03 02:33:47.598625 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.598603 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-zsswz"] Nov 03 02:33:47.707443 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.707405 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4"] Nov 03 02:33:47.710491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:47.710470 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-4d7b4"] Nov 03 02:33:48.079359 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:48.079268 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2"] Nov 03 02:33:48.097260 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:48.097221 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-kdds2"] Nov 03 02:33:48.101357 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:48.101296 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl"] Nov 03 02:33:48.104343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:48.104304 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-x82xl"] Nov 03 02:33:48.214903 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:48.214876 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb" Nov 03 02:33:48.215071 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:48.214956 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw" Nov 03 02:33:48.277715 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:48.277659 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:48.437238 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:48.437178 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:48.443954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:48.443928 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21beedd4-fac4-4b92-9cf8-c2acb85422f4" path="/var/lib/kubelet/pods/21beedd4-fac4-4b92-9cf8-c2acb85422f4/volumes" Nov 03 02:33:48.444462 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:48.444446 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad2720fd-aa19-4213-a377-c98474134c93" path="/var/lib/kubelet/pods/ad2720fd-aa19-4213-a377-c98474134c93/volumes" Nov 03 02:33:48.445009 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:48.444994 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4fd375-fcd1-4665-8a2e-89a67bb9dc43" path="/var/lib/kubelet/pods/da4fd375-fcd1-4665-8a2e-89a67bb9dc43/volumes" Nov 03 02:33:48.445799 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:48.445782 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1bab284-6e6d-4ace-a4af-65fd25483815" path="/var/lib/kubelet/pods/e1bab284-6e6d-4ace-a4af-65fd25483815/volumes" Nov 03 02:33:48.531513 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:48.531485 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 03 02:33:48.531714 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:48.531538 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" podUID="a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 03 02:33:48.596323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:48.596294 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:49.531063 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:49.531023 5463 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 03 02:33:49.531578 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:49.531078 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" podUID="a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 03 02:33:49.731703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:49.731666 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:49.892559 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:49.892478 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:50.071875 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:50.071829 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:50.599633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:50.599600 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:50.600048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:50.599999 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:50.600954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:50.600932 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:33:52.890645 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:52.890612 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:53.074141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.074091 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:53.293936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.293897 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79"] Nov 03 02:33:53.294834 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.294815 5463 cpu_manager.go:414] "containerMap: removing container" podUID="ad2720fd-aa19-4213-a377-c98474134c93" containerName="istio-proxy" Nov 03 02:33:53.294891 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.294838 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad2720fd-aa19-4213-a377-c98474134c93" containerName="istio-proxy" Nov 03 02:33:53.294944 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.294893 5463 cpu_manager.go:414] "containerMap: removing container" podUID="e1bab284-6e6d-4ace-a4af-65fd25483815" containerName="istio-proxy" Nov 03 02:33:53.294944 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.294902 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1bab284-6e6d-4ace-a4af-65fd25483815" containerName="istio-proxy" Nov 03 02:33:53.294944 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.294921 5463 cpu_manager.go:414] "containerMap: removing container" podUID="da4fd375-fcd1-4665-8a2e-89a67bb9dc43" containerName="istio-proxy" Nov 03 02:33:53.294944 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.294928 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4fd375-fcd1-4665-8a2e-89a67bb9dc43" containerName="istio-proxy" Nov 03 02:33:53.294944 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.294938 5463 cpu_manager.go:414] "containerMap: removing container" podUID="21beedd4-fac4-4b92-9cf8-c2acb85422f4" containerName="istio-proxy" Nov 03 02:33:53.295101 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.294946 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21beedd4-fac4-4b92-9cf8-c2acb85422f4" containerName="istio-proxy" Nov 03 02:33:53.295101 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.294966 5463 cpu_manager.go:414] "containerMap: removing container" podUID="da4fd375-fcd1-4665-8a2e-89a67bb9dc43" containerName="istio-proxy" Nov 03 02:33:53.295101 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.294973 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4fd375-fcd1-4665-8a2e-89a67bb9dc43" containerName="istio-proxy" Nov 03 02:33:53.295214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.295121 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad2720fd-aa19-4213-a377-c98474134c93" containerName="istio-proxy" Nov 03 02:33:53.295214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.295140 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1bab284-6e6d-4ace-a4af-65fd25483815" containerName="istio-proxy" Nov 03 02:33:53.295214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.295152 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4fd375-fcd1-4665-8a2e-89a67bb9dc43" containerName="istio-proxy" Nov 03 02:33:53.295214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.295161 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="21beedd4-fac4-4b92-9cf8-c2acb85422f4" containerName="istio-proxy" Nov 03 02:33:53.295214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.295172 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4fd375-fcd1-4665-8a2e-89a67bb9dc43" containerName="istio-proxy" Nov 03 02:33:53.301301 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.301280 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79" Nov 03 02:33:53.343378 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.343352 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79\" (UID: \"98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79" Nov 03 02:33:53.343498 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.343482 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2h7h\" (UniqueName: \"kubernetes.io/projected/98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7-kube-api-access-f2h7h\") pod \"payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79\" (UID: \"98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79" Nov 03 02:33:53.347007 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.346983 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79"] Nov 03 02:33:53.412213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.398505 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f954db\"" Nov 03 02:33:53.412213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.398640 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f954db-dockercfg-m6rbh\"" Nov 03 02:33:53.459102 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.459052 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf"] Nov 03 02:33:53.463246 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.463211 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2h7h\" (UniqueName: \"kubernetes.io/projected/98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7-kube-api-access-f2h7h\") pod \"payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79\" (UID: \"98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79" Nov 03 02:33:53.463517 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.463478 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79\" (UID: \"98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79" Nov 03 02:33:53.471250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.464583 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79\" (UID: \"98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79" Nov 03 02:33:53.471250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.465978 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf" Nov 03 02:33:53.508595 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.508436 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf"] Nov 03 02:33:53.520823 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.520792 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2h7h\" (UniqueName: \"kubernetes.io/projected/98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7-kube-api-access-f2h7h\") pod \"payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79\" (UID: \"98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79" Nov 03 02:33:53.565550 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.565466 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjq7z\" (UniqueName: \"kubernetes.io/projected/f2251999-6e8a-453b-92d2-c914e4ca59b7-kube-api-access-bjq7z\") pod \"payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf\" (UID: \"f2251999-6e8a-453b-92d2-c914e4ca59b7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf" Nov 03 02:33:53.565688 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.565626 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2251999-6e8a-453b-92d2-c914e4ca59b7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf\" (UID: \"f2251999-6e8a-453b-92d2-c914e4ca59b7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf" Nov 03 02:33:53.616081 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.616045 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79" Nov 03 02:33:53.667403 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.667371 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjq7z\" (UniqueName: \"kubernetes.io/projected/f2251999-6e8a-453b-92d2-c914e4ca59b7-kube-api-access-bjq7z\") pod \"payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf\" (UID: \"f2251999-6e8a-453b-92d2-c914e4ca59b7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf" Nov 03 02:33:53.667554 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.667489 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2251999-6e8a-453b-92d2-c914e4ca59b7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf\" (UID: \"f2251999-6e8a-453b-92d2-c914e4ca59b7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf" Nov 03 02:33:53.668333 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.668311 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2251999-6e8a-453b-92d2-c914e4ca59b7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf\" (UID: \"f2251999-6e8a-453b-92d2-c914e4ca59b7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf" Nov 03 02:33:53.724851 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.724816 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjq7z\" (UniqueName: \"kubernetes.io/projected/f2251999-6e8a-453b-92d2-c914e4ca59b7-kube-api-access-bjq7z\") pod \"payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf\" (UID: \"f2251999-6e8a-453b-92d2-c914e4ca59b7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf" Nov 03 02:33:53.810568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:53.810539 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf" Nov 03 02:33:54.000027 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:33:53.999998 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a4bbfb_fb5a_44f1_8cbe_bf808d4b1db7.slice/crio-2a7ce686a64292c2553df637e14090b5664b9b80d4dc15563cc48ff3465a1e21 WatchSource:0}: Error finding container 2a7ce686a64292c2553df637e14090b5664b9b80d4dc15563cc48ff3465a1e21: Status 404 returned error can't find the container with id 2a7ce686a64292c2553df637e14090b5664b9b80d4dc15563cc48ff3465a1e21 Nov 03 02:33:54.001746 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:54.001721 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79"] Nov 03 02:33:54.189547 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:33:54.189512 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2251999_6e8a_453b_92d2_c914e4ca59b7.slice/crio-fbcd7521aa61d896bc2bb5e52fbaec52d3c291f146b80eca99321017ca84d359 WatchSource:0}: Error finding container fbcd7521aa61d896bc2bb5e52fbaec52d3c291f146b80eca99321017ca84d359: Status 404 returned error can't find the container with id fbcd7521aa61d896bc2bb5e52fbaec52d3c291f146b80eca99321017ca84d359 Nov 03 02:33:54.191146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:54.191121 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf"] Nov 03 02:33:54.495882 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:54.495787 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf" event={"ID":"f2251999-6e8a-453b-92d2-c914e4ca59b7","Type":"ContainerStarted","Data":"194fc04144a3588832d877d871dc641c47f95df40eaec61d72c0135e4b466a92"} Nov 03 02:33:54.495882 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:54.495841 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf" event={"ID":"f2251999-6e8a-453b-92d2-c914e4ca59b7","Type":"ContainerStarted","Data":"fbcd7521aa61d896bc2bb5e52fbaec52d3c291f146b80eca99321017ca84d359"} Nov 03 02:33:54.496113 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:54.495880 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf" Nov 03 02:33:54.497757 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:54.497730 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79" event={"ID":"98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7","Type":"ContainerStarted","Data":"5f0cb59c0fd1b33583ec62b60e471344e8d12bacb0119d590f0429da5cacbcb2"} Nov 03 02:33:54.497873 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:54.497777 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79" event={"ID":"98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7","Type":"ContainerStarted","Data":"2a7ce686a64292c2553df637e14090b5664b9b80d4dc15563cc48ff3465a1e21"} Nov 03 02:33:54.497873 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:54.497822 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79" Nov 03 02:33:54.611061 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:54.611015 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf" podStartSLOduration=1.611002388 podStartE2EDuration="1.611002388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:54.609333836 +0000 UTC m=+2516.293442990" watchObservedRunningTime="2026-11-03 02:33:54.611002388 +0000 UTC m=+2516.295111593" Nov 03 02:33:54.725584 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:54.725518 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79" podStartSLOduration=1.725498284 podStartE2EDuration="1.725498284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:54.722705542 +0000 UTC m=+2516.406814700" watchObservedRunningTime="2026-11-03 02:33:54.725498284 +0000 UTC m=+2516.409607441" Nov 03 02:33:55.171867 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:55.171829 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:59.918584 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:33:59.918547 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:04.734120 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:04.734088 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:05.508068 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:05.508037 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79" Nov 03 02:34:05.508292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:05.508109 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf" Nov 03 02:34:09.374787 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:09.374699 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:09.532666 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:09.532635 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:09.821077 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:09.820979 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:13.007244 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:34:13.007180 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:34:14.072125 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.072086 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9"] Nov 03 02:34:14.081310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.081289 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" Nov 03 02:34:14.115878 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.115849 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9"] Nov 03 02:34:14.176220 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.174078 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-f735f4-serving-cert\"" Nov 03 02:34:14.246547 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.246510 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rksj\" (UniqueName: \"kubernetes.io/projected/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-kube-api-access-2rksj\") pod \"maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9\" (UID: \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" Nov 03 02:34:14.246718 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.246595 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9\" (UID: \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" Nov 03 02:34:14.246718 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.246631 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9\" (UID: \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" Nov 03 02:34:14.246802 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.246723 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9\" (UID: \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" Nov 03 02:34:14.273834 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.273767 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:14.347932 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.347829 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rksj\" (UniqueName: \"kubernetes.io/projected/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-kube-api-access-2rksj\") pod \"maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9\" (UID: \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" Nov 03 02:34:14.348711 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.348682 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9\" (UID: \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" Nov 03 02:34:14.368332 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.368298 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9\" (UID: \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" Nov 03 02:34:14.368657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.368638 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9\" (UID: \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" Nov 03 02:34:14.369372 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:34:14.369346 5463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-serving-cert: secret "maas-api-e2e-shared-a-w2-f735f4-serving-cert" not found Nov 03 02:34:14.369584 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:34:14.369570 5463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-maas-api-tls podName:d6da7ddb-bf04-456f-9821-dfb9d517e0b6 nodeName:}" failed. No retries permitted until 2026-11-03 02:34:14.869547767 +0000 UTC m=+2536.553656920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-maas-api-tls") pod "maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" (UID: "d6da7ddb-bf04-456f-9821-dfb9d517e0b6") : secret "maas-api-e2e-shared-a-w2-f735f4-serving-cert" not found Nov 03 02:34:14.369912 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.369874 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9\" (UID: \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" Nov 03 02:34:14.381985 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.381950 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9\" (UID: \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" Nov 03 02:34:14.406495 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.406469 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rksj\" (UniqueName: \"kubernetes.io/projected/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-kube-api-access-2rksj\") pod \"maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9\" (UID: \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" Nov 03 02:34:14.426715 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.426689 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:14.581221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.580723 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:14.887065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.887011 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9\" (UID: \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" Nov 03 02:34:14.890637 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.890612 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9\" (UID: \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" Nov 03 02:34:14.996507 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:14.996475 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" Nov 03 02:34:15.402277 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:34:15.402244 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6da7ddb_bf04_456f_9821_dfb9d517e0b6.slice/crio-86f93b06035c56d04e9f030a5c992efdb70fb66a9494210c6d277b9790e74858 WatchSource:0}: Error finding container 86f93b06035c56d04e9f030a5c992efdb70fb66a9494210c6d277b9790e74858: Status 404 returned error can't find the container with id 86f93b06035c56d04e9f030a5c992efdb70fb66a9494210c6d277b9790e74858 Nov 03 02:34:15.406004 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:15.405980 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9"] Nov 03 02:34:15.681704 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:15.681618 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" event={"ID":"d6da7ddb-bf04-456f-9821-dfb9d517e0b6","Type":"ContainerStarted","Data":"86f93b06035c56d04e9f030a5c992efdb70fb66a9494210c6d277b9790e74858"} Nov 03 02:34:16.690637 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:16.690601 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" event={"ID":"d6da7ddb-bf04-456f-9821-dfb9d517e0b6","Type":"ContainerStarted","Data":"d5b0f3493ee25a38d19d119985fae1178bd1db0c86c0866e427dd5b95176458a"} Nov 03 02:34:16.691106 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:16.690683 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" Nov 03 02:34:16.809020 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:16.808970 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" podStartSLOduration=1.816257958 podStartE2EDuration="2.808955411s" totalImagesPullingTime="992.697453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:34:16.805287694 +0000 UTC m=+2538.489396846" watchObservedRunningTime="2026-11-03 02:34:16.808955411 +0000 UTC m=+2538.493064562" Nov 03 02:34:17.238153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:17.238122 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:17.404815 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:17.404779 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:18.677334 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:18.677298 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:34:18.677801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:18.677374 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" podUID="04066f9c-931a-4c65-a714-57270768f0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:34:19.216355 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:19.216321 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:20.726698 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:20.726663 5463 generic.go:356] "Generic (PLEG): container finished" podID="04066f9c-931a-4c65-a714-57270768f0e5" containerID="9188fa87fe3fa573942fbd56639310a8913d4380d4bd2f47c085360a8817826f" exitCode=0 Nov 03 02:34:20.727292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:20.726733 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" event={"ID":"04066f9c-931a-4c65-a714-57270768f0e5","Type":"ContainerDied","Data":"9188fa87fe3fa573942fbd56639310a8913d4380d4bd2f47c085360a8817826f"} Nov 03 02:34:20.727292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:20.727147 5463 scope.go:122] "RemoveContainer" containerID="9188fa87fe3fa573942fbd56639310a8913d4380d4bd2f47c085360a8817826f" Nov 03 02:34:20.728691 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:20.728661 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:20.728791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:20.728726 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:20.728791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:20.728769 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:21.736704 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:21.736668 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" event={"ID":"04066f9c-931a-4c65-a714-57270768f0e5","Type":"ContainerStarted","Data":"dd45e4e9d6e4d5e77487e402bfedcf23814c8cdfc921a9e1c099a3dcd7e12867"} Nov 03 02:34:22.312257 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:22.312228 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:34:22.312462 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:22.312439 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Nov 03 02:34:22.312552 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:22.312495 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" podUID="04066f9c-931a-4c65-a714-57270768f0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Nov 03 02:34:22.706367 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:22.706341 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" Nov 03 02:34:23.312242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:23.312173 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Nov 03 02:34:23.312638 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:23.312278 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" podUID="04066f9c-931a-4c65-a714-57270768f0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Nov 03 02:34:23.929618 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:23.929583 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:24.312482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:24.312399 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Nov 03 02:34:24.312482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:24.312463 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" podUID="04066f9c-931a-4c65-a714-57270768f0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Nov 03 02:34:25.312970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:25.312922 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Nov 03 02:34:25.313388 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:25.312991 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" podUID="04066f9c-931a-4c65-a714-57270768f0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Nov 03 02:34:26.312045 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:26.312013 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Nov 03 02:34:26.312264 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:26.312065 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" podUID="04066f9c-931a-4c65-a714-57270768f0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Nov 03 02:34:27.316223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:27.316172 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:34:27.316648 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:27.316636 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:34:27.317432 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:27.317411 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 02:34:28.626673 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:28.626633 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:33.413035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:33.412983 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:36.268184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:36.268146 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:36.703676 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:36.703631 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:37.181950 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.181914 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54"] Nov 03 02:34:37.185469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.185444 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.214956 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.214917 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-envoy\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.215127 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.214971 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-token\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.215127 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.215047 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-workload-certs\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.215127 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.215107 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-credential-socket\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.215342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.215236 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-podinfo\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.215670 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.215627 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-workload-socket\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.215783 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.215764 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7tpf\" (UniqueName: \"kubernetes.io/projected/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-kube-api-access-d7tpf\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.215914 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.215886 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-data\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.216017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.215997 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istiod-ca-cert\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.240012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.239709 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-7273af-openshift-default-dockercfg-jp82l\"" Nov 03 02:34:37.293470 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.293434 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54"] Nov 03 02:34:37.297827 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.296965 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:37.317374 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.316906 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istiod-ca-cert\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.317374 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.316956 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-envoy\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.317374 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.316988 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-token\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.318703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.317723 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-workload-certs\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.318703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.317800 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-credential-socket\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.318703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.317880 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-podinfo\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.318703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.317950 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-workload-socket\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.318703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.318025 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7tpf\" (UniqueName: \"kubernetes.io/projected/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-kube-api-access-d7tpf\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.318703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.318092 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-data\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.318703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.318097 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-workload-certs\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.318703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.318161 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istiod-ca-cert\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.318703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.318451 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-data\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.318703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.318517 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-credential-socket\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.319479 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.319452 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-workload-socket\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.338893 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.322488 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-envoy\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.338893 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.333201 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-podinfo\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.422250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.408982 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7tpf\" (UniqueName: \"kubernetes.io/projected/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-kube-api-access-d7tpf\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.422250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.413105 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-token\") pod \"e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.513264 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.503818 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:37.522457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.515276 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:37.957619 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:34:37.957577 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b9bf51_7d4e_4a3d_9c86_d19e80745d2a.slice/crio-cdddfa897e7ad3fd41b4c7eb9be8ecd3d2fe4b6c014a3f47b4c47e268a0c6e69 WatchSource:0}: Error finding container cdddfa897e7ad3fd41b4c7eb9be8ecd3d2fe4b6c014a3f47b4c47e268a0c6e69: Status 404 returned error can't find the container with id cdddfa897e7ad3fd41b4c7eb9be8ecd3d2fe4b6c014a3f47b4c47e268a0c6e69 Nov 03 02:34:37.960666 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.960625 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:37.960792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.960715 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:37.960792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.960750 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:37.962341 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:37.962280 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54"] Nov 03 02:34:38.409582 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:38.409545 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:38.559220 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:38.559135 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:38.711934 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:38.711860 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:38.957527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:38.957484 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" event={"ID":"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a","Type":"ContainerStarted","Data":"7f7d2be25db222b2001948ef40d630d02d2bd4dfb47b2693fad9596345175625"} Nov 03 02:34:38.957731 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:38.957569 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" event={"ID":"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a","Type":"ContainerStarted","Data":"cdddfa897e7ad3fd41b4c7eb9be8ecd3d2fe4b6c014a3f47b4c47e268a0c6e69"} Nov 03 02:34:39.131138 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:39.131021 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" podStartSLOduration=2.131001841 podStartE2EDuration="2.131001841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:34:39.125448447 +0000 UTC m=+2560.809557615" watchObservedRunningTime="2026-11-03 02:34:39.131001841 +0000 UTC m=+2560.815110998" Nov 03 02:34:39.514083 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:39.514043 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:39.531147 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:39.520826 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:40.433902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:39.967688 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:40.433902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:39.968750 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:34:40.472127 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.472097 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s"] Nov 03 02:34:40.496403 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.481617 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.584173 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.584124 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.584695 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.584357 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.584695 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.584407 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3926283-2107-4c88-8c94-744886250f75-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.584695 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.584530 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3926283-2107-4c88-8c94-744886250f75-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.584695 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.584577 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3926283-2107-4c88-8c94-744886250f75-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.584695 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.584629 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whkdr\" (UniqueName: \"kubernetes.io/projected/e3926283-2107-4c88-8c94-744886250f75-kube-api-access-whkdr\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.584695 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.584687 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.585071 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.584841 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.585071 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.584897 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.675771 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.675737 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s"] Nov 03 02:34:40.685783 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.685697 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.685783 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.685737 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3926283-2107-4c88-8c94-744886250f75-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.686018 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.685791 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3926283-2107-4c88-8c94-744886250f75-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.686018 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.685831 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3926283-2107-4c88-8c94-744886250f75-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.686018 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.685857 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whkdr\" (UniqueName: \"kubernetes.io/projected/e3926283-2107-4c88-8c94-744886250f75-kube-api-access-whkdr\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.686018 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.685896 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.686018 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.686010 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.686328 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.686055 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.686328 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.686121 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.686328 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.686131 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.686486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.686418 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.686486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.686423 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.686589 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.686496 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.686937 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.686912 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3926283-2107-4c88-8c94-744886250f75-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.688463 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.688439 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.699434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.699407 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3926283-2107-4c88-8c94-744886250f75-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.742669 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.742639 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whkdr\" (UniqueName: \"kubernetes.io/projected/e3926283-2107-4c88-8c94-744886250f75-kube-api-access-whkdr\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.742774 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.742722 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3926283-2107-4c88-8c94-744886250f75-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.815892 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.815866 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:40.975345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.975265 5463 generic.go:356] "Generic (PLEG): container finished" podID="2d2157b2-209f-4ed7-9aab-a79fbe023d82" containerID="92bf2fd408a99e0a6e1de092dee3d681167da3227d30119d33c6a565c74042b6" exitCode=0 Nov 03 02:34:40.975513 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.975341 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" event={"ID":"2d2157b2-209f-4ed7-9aab-a79fbe023d82","Type":"ContainerDied","Data":"92bf2fd408a99e0a6e1de092dee3d681167da3227d30119d33c6a565c74042b6"} Nov 03 02:34:40.975807 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.975781 5463 scope.go:122] "RemoveContainer" containerID="92bf2fd408a99e0a6e1de092dee3d681167da3227d30119d33c6a565c74042b6" Nov 03 02:34:40.977310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.977288 5463 generic.go:356] "Generic (PLEG): container finished" podID="5ddcf4ff-6e4f-45cd-81a8-8b283429291a" containerID="8f57f4bda498318d5b27f65f2842ab61418b7f4cb5577faa26326b5bc9ee2154" exitCode=0 Nov 03 02:34:40.977425 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.977361 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" event={"ID":"5ddcf4ff-6e4f-45cd-81a8-8b283429291a","Type":"ContainerDied","Data":"8f57f4bda498318d5b27f65f2842ab61418b7f4cb5577faa26326b5bc9ee2154"} Nov 03 02:34:40.977476 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.977359 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:40.977535 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.977513 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:40.977590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.977573 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:40.977800 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.977785 5463 scope.go:122] "RemoveContainer" containerID="8f57f4bda498318d5b27f65f2842ab61418b7f4cb5577faa26326b5bc9ee2154" Nov 03 02:34:40.979114 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.979081 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:40.979238 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.979146 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:40.979238 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:40.979178 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:41.259998 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:41.259879 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s"] Nov 03 02:34:41.270566 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:41.270532 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:41.270750 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:41.270612 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:41.270750 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:41.270645 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:41.947109 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:41.947074 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:34:41.947820 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:41.947798 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:34:41.987232 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:41.987173 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:42.032249 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:42.031107 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" event={"ID":"2d2157b2-209f-4ed7-9aab-a79fbe023d82","Type":"ContainerStarted","Data":"7326da7a8920c6ba29d5fe4fd2dcaea9e1ac8dd7e3a2d9d30881f207c375a047"} Nov 03 02:34:42.035970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:42.035938 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" event={"ID":"5ddcf4ff-6e4f-45cd-81a8-8b283429291a","Type":"ContainerStarted","Data":"ef5a7c160b9241132bb442aa14c975134886b8f3263669e78c7ef35c43d8b634"} Nov 03 02:34:42.046662 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:42.046607 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" event={"ID":"e3926283-2107-4c88-8c94-744886250f75","Type":"ContainerStarted","Data":"2cb1b26a5c64df734385ed39b1983fc702543dfb919c676a7c9d55fc5d468537"} Nov 03 02:34:42.046993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:42.046971 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" event={"ID":"e3926283-2107-4c88-8c94-744886250f75","Type":"ContainerStarted","Data":"a1b7843fe90a95cc01f0aad60eebc280f91499c44843ccb6d7a4ae82e1de464a"} Nov 03 02:34:42.109361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:42.109325 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:34:42.110224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:42.110180 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 03 02:34:42.110436 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:42.110409 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" podUID="5ddcf4ff-6e4f-45cd-81a8-8b283429291a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 03 02:34:42.233363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:42.233280 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:34:42.234310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:42.234283 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Nov 03 02:34:42.234451 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:42.234349 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" podUID="2d2157b2-209f-4ed7-9aab-a79fbe023d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Nov 03 02:34:42.242809 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:42.242780 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:42.324040 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:42.323975 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" podStartSLOduration=2.323955499 podStartE2EDuration="2.323955499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:34:42.321331123 +0000 UTC m=+2564.005440291" watchObservedRunningTime="2026-11-03 02:34:42.323955499 +0000 UTC m=+2564.008064656" Nov 03 02:34:42.408860 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:42.408820 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:42.821216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:42.819703 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:42.821216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:42.820496 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Nov 03 02:34:42.821216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:42.820541 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" podUID="e3926283-2107-4c88-8c94-744886250f75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Nov 03 02:34:43.104575 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:43.104472 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 03 02:34:43.105166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:43.105119 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" podUID="5ddcf4ff-6e4f-45cd-81a8-8b283429291a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 03 02:34:43.146018 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:34:43.145969 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:34:43.232217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:43.231633 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Nov 03 02:34:43.232217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:43.231694 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" podUID="2d2157b2-209f-4ed7-9aab-a79fbe023d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Nov 03 02:34:43.818228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:43.817823 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Nov 03 02:34:43.818228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:43.817896 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" podUID="e3926283-2107-4c88-8c94-744886250f75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Nov 03 02:34:43.883135 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:43.883096 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:44.122274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:44.120406 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 03 02:34:44.122274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:44.120461 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" podUID="5ddcf4ff-6e4f-45cd-81a8-8b283429291a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 03 02:34:44.231073 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:44.231033 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Nov 03 02:34:44.231273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:44.231088 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" podUID="2d2157b2-209f-4ed7-9aab-a79fbe023d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Nov 03 02:34:44.817214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:44.816553 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Nov 03 02:34:44.817214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:44.816614 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" podUID="e3926283-2107-4c88-8c94-744886250f75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Nov 03 02:34:45.110271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:45.109505 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 03 02:34:45.110271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:45.109579 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" podUID="5ddcf4ff-6e4f-45cd-81a8-8b283429291a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 03 02:34:45.233215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:45.232605 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Nov 03 02:34:45.233215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:45.232665 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" podUID="2d2157b2-209f-4ed7-9aab-a79fbe023d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Nov 03 02:34:45.831217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:45.827374 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Nov 03 02:34:45.831217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:45.827436 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" podUID="e3926283-2107-4c88-8c94-744886250f75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Nov 03 02:34:46.109866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:46.109773 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 03 02:34:46.110618 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:46.110588 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" podUID="5ddcf4ff-6e4f-45cd-81a8-8b283429291a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 03 02:34:46.236217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:46.231731 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Nov 03 02:34:46.236217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:46.231810 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" podUID="2d2157b2-209f-4ed7-9aab-a79fbe023d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Nov 03 02:34:46.821215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:46.820933 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Nov 03 02:34:46.821215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:46.820996 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" podUID="e3926283-2107-4c88-8c94-744886250f75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Nov 03 02:34:47.104492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:47.104391 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 03 02:34:47.104774 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:47.104745 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" podUID="5ddcf4ff-6e4f-45cd-81a8-8b283429291a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 03 02:34:47.232357 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:47.232306 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Nov 03 02:34:47.232550 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:47.232404 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" podUID="2d2157b2-209f-4ed7-9aab-a79fbe023d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Nov 03 02:34:47.447100 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:47.446904 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:47.656665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:47.656615 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:47.855921 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:47.855681 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h"] Nov 03 02:34:47.860758 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:47.860252 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h" Nov 03 02:34:47.917224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:47.912634 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:47.917224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:47.913910 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:47.919113 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:47.919059 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:34:47.919756 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:47.919735 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h"] Nov 03 02:34:47.988212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:47.958551 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428a7cac-d801-42a2-8e40-eb7a1e4c9657-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h\" (UID: \"428a7cac-d801-42a2-8e40-eb7a1e4c9657\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h" Nov 03 02:34:48.012215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:47.974570 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-7273af-dockercfg-sh2pn\"" Nov 03 02:34:48.012215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:47.974633 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7273af\"" Nov 03 02:34:48.012215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:47.993561 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgmql\" (UniqueName: \"kubernetes.io/projected/428a7cac-d801-42a2-8e40-eb7a1e4c9657-kube-api-access-xgmql\") pod \"payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h\" (UID: \"428a7cac-d801-42a2-8e40-eb7a1e4c9657\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h" Nov 03 02:34:48.059055 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.059022 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k"] Nov 03 02:34:48.065395 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.065359 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k"] Nov 03 02:34:48.066320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.066291 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k" Nov 03 02:34:48.095017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.094988 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428a7cac-d801-42a2-8e40-eb7a1e4c9657-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h\" (UID: \"428a7cac-d801-42a2-8e40-eb7a1e4c9657\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h" Nov 03 02:34:48.095170 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.095044 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgmql\" (UniqueName: \"kubernetes.io/projected/428a7cac-d801-42a2-8e40-eb7a1e4c9657-kube-api-access-xgmql\") pod \"payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h\" (UID: \"428a7cac-d801-42a2-8e40-eb7a1e4c9657\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h" Nov 03 02:34:48.095807 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.095787 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428a7cac-d801-42a2-8e40-eb7a1e4c9657-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h\" (UID: \"428a7cac-d801-42a2-8e40-eb7a1e4c9657\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h" Nov 03 02:34:48.108108 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.108056 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:34:48.108417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.108395 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:34:48.109089 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.109072 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 02:34:48.152457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.152425 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgmql\" (UniqueName: \"kubernetes.io/projected/428a7cac-d801-42a2-8e40-eb7a1e4c9657-kube-api-access-xgmql\") pod \"payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h\" (UID: \"428a7cac-d801-42a2-8e40-eb7a1e4c9657\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h" Nov 03 02:34:48.178488 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.178460 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h" Nov 03 02:34:48.234503 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.234465 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8390f42a-ae1c-4a3d-94ee-8fade238051e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k\" (UID: \"8390f42a-ae1c-4a3d-94ee-8fade238051e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k" Nov 03 02:34:48.234923 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.234903 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tjmk\" (UniqueName: \"kubernetes.io/projected/8390f42a-ae1c-4a3d-94ee-8fade238051e-kube-api-access-9tjmk\") pod \"payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k\" (UID: \"8390f42a-ae1c-4a3d-94ee-8fade238051e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k" Nov 03 02:34:48.263744 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.263689 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:34:48.264590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.264561 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:34:48.265355 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.265328 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:34:48.338039 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.337758 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8390f42a-ae1c-4a3d-94ee-8fade238051e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k\" (UID: \"8390f42a-ae1c-4a3d-94ee-8fade238051e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k" Nov 03 02:34:48.343355 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.339607 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tjmk\" (UniqueName: \"kubernetes.io/projected/8390f42a-ae1c-4a3d-94ee-8fade238051e-kube-api-access-9tjmk\") pod \"payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k\" (UID: \"8390f42a-ae1c-4a3d-94ee-8fade238051e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k" Nov 03 02:34:48.343355 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.341017 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8390f42a-ae1c-4a3d-94ee-8fade238051e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k\" (UID: \"8390f42a-ae1c-4a3d-94ee-8fade238051e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k" Nov 03 02:34:48.402134 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.402101 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tjmk\" (UniqueName: \"kubernetes.io/projected/8390f42a-ae1c-4a3d-94ee-8fade238051e-kube-api-access-9tjmk\") pod \"payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k\" (UID: \"8390f42a-ae1c-4a3d-94ee-8fade238051e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k" Nov 03 02:34:48.663395 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:34:48.663363 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428a7cac_d801_42a2_8e40_eb7a1e4c9657.slice/crio-6abfbe5dd9b1875f6c60359f7172a26a23394c657fd7ab50966931930a0e4f43 WatchSource:0}: Error finding container 6abfbe5dd9b1875f6c60359f7172a26a23394c657fd7ab50966931930a0e4f43: Status 404 returned error can't find the container with id 6abfbe5dd9b1875f6c60359f7172a26a23394c657fd7ab50966931930a0e4f43 Nov 03 02:34:48.666524 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.666501 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h"] Nov 03 02:34:48.693888 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:48.693865 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k" Nov 03 02:34:49.037362 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:49.037271 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:49.077137 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:49.077101 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k"] Nov 03 02:34:49.275302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:49.275252 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h" event={"ID":"428a7cac-d801-42a2-8e40-eb7a1e4c9657","Type":"ContainerStarted","Data":"d2846a5de06b801f540f138f981cf88859644c20bd63178c1742d4040a212439"} Nov 03 02:34:49.275507 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:49.275356 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h" event={"ID":"428a7cac-d801-42a2-8e40-eb7a1e4c9657","Type":"ContainerStarted","Data":"6abfbe5dd9b1875f6c60359f7172a26a23394c657fd7ab50966931930a0e4f43"} Nov 03 02:34:49.275507 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:49.275387 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h" Nov 03 02:34:49.277348 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:49.277315 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k" event={"ID":"8390f42a-ae1c-4a3d-94ee-8fade238051e","Type":"ContainerStarted","Data":"ec3ead13071c6d94c11614c610dfe594441c992eee877d71d48f48f291d4436d"} Nov 03 02:34:49.277511 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:49.277370 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k" event={"ID":"8390f42a-ae1c-4a3d-94ee-8fade238051e","Type":"ContainerStarted","Data":"0ea8e841a323dc344f68f2f4f765a570ce5c5dced539e801a41528567b12010d"} Nov 03 02:34:49.277511 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:49.277467 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k" Nov 03 02:34:49.412879 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:49.393850 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h" podStartSLOduration=2.393833185 podStartE2EDuration="2.393833185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:34:49.391559263 +0000 UTC m=+2571.075668419" watchObservedRunningTime="2026-11-03 02:34:49.393833185 +0000 UTC m=+2571.077942342" Nov 03 02:34:49.508527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:49.508462 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k" podStartSLOduration=2.508420184 podStartE2EDuration="2.508420184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:34:49.504720551 +0000 UTC m=+2571.188829707" watchObservedRunningTime="2026-11-03 02:34:49.508420184 +0000 UTC m=+2571.192529336" Nov 03 02:34:51.838204 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:51.838158 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:52.301206 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:52.301168 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:53.137244 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.137186 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w"] Nov 03 02:34:53.140811 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.140794 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.194244 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.194173 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-78b5c5-openshift-default-dockercfg-5kl7k\"" Nov 03 02:34:53.244570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.244539 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-workload-certs\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.244768 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.244587 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-credential-socket\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.244768 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.244616 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-envoy\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.244768 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.244680 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-data\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.244768 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.244728 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-podinfo\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.245030 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.244815 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istiod-ca-cert\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.245030 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.244885 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2vgt\" (UniqueName: \"kubernetes.io/projected/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-kube-api-access-x2vgt\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.245030 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.244953 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-workload-socket\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.245030 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.245014 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-token\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.248481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.248452 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w"] Nov 03 02:34:53.249911 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.249878 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:53.346694 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.346650 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-workload-certs\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.346911 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.346713 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-credential-socket\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.346911 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.346737 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-envoy\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.346911 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.346753 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-data\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.346911 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.346776 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-podinfo\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.346911 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.346838 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istiod-ca-cert\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.347240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.346930 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vgt\" (UniqueName: \"kubernetes.io/projected/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-kube-api-access-x2vgt\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.347240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.346999 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-workload-socket\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.347240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.347069 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-token\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.347240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.347127 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-workload-certs\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.347240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.347219 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-data\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.347535 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.347511 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-credential-socket\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.347696 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.347667 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istiod-ca-cert\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.347849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.347825 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-workload-socket\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.349439 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.349415 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-envoy\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.349857 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.349832 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-podinfo\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.404753 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.404726 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-token\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.404936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.404918 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2vgt\" (UniqueName: \"kubernetes.io/projected/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-kube-api-access-x2vgt\") pod \"e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.455285 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.455263 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:53.509779 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.509740 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:53.893077 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:34:53.893048 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8b0d5f_94b6_439a_bbb0_35fb828131dc.slice/crio-3a1c4961c696a147e455fd24ecb613b00c2513583aa973b6313117fa85976d66 WatchSource:0}: Error finding container 3a1c4961c696a147e455fd24ecb613b00c2513583aa973b6313117fa85976d66: Status 404 returned error can't find the container with id 3a1c4961c696a147e455fd24ecb613b00c2513583aa973b6313117fa85976d66 Nov 03 02:34:53.895908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.895882 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w"] Nov 03 02:34:53.896416 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.896386 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:53.896491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.896452 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:53.896491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:53.896484 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:34:54.314857 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:54.314819 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:54.330611 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:54.330576 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" event={"ID":"8f8b0d5f-94b6-439a-bbb0-35fb828131dc","Type":"ContainerStarted","Data":"fbaeb2432e1d7a5bf0a9433268d8a332624e3c7d5158bfbb054a60d28513baab"} Nov 03 02:34:54.330791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:54.330639 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" event={"ID":"8f8b0d5f-94b6-439a-bbb0-35fb828131dc","Type":"ContainerStarted","Data":"3a1c4961c696a147e455fd24ecb613b00c2513583aa973b6313117fa85976d66"} Nov 03 02:34:54.450931 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:54.450879 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" podStartSLOduration=1.450864305 podStartE2EDuration="1.450864305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:34:54.44703121 +0000 UTC m=+2576.131140366" watchObservedRunningTime="2026-11-03 02:34:54.450864305 +0000 UTC m=+2576.134973461" Nov 03 02:34:54.455735 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:54.455711 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:54.461585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:54.461562 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:55.338693 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:55.338669 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:55.339534 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:55.339514 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:34:57.256004 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:57.255966 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:57.418541 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:57.418505 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:57.605939 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:57.605815 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:59.527567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:34:59.527531 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:00.287750 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:00.287720 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k" Nov 03 02:35:00.287932 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:00.287794 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h" Nov 03 02:35:00.533905 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:00.533866 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:00.700700 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:00.700662 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:01.244033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:01.243995 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv"] Nov 03 02:35:01.276676 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:01.276642 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv" Nov 03 02:35:01.298490 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:01.298462 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv"] Nov 03 02:35:01.324333 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:01.324298 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnsnx\" (UniqueName: \"kubernetes.io/projected/ed76fc49-8f0a-4a64-a273-01cb7a2d612b-kube-api-access-dnsnx\") pod \"payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv\" (UID: \"ed76fc49-8f0a-4a64-a273-01cb7a2d612b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv" Nov 03 02:35:01.324703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:01.324681 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed76fc49-8f0a-4a64-a273-01cb7a2d612b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv\" (UID: \"ed76fc49-8f0a-4a64-a273-01cb7a2d612b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv" Nov 03 02:35:01.345216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:01.345175 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-78b5c5\"" Nov 03 02:35:01.345310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:01.345224 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-78b5c5-dockercfg-6pxlq\"" Nov 03 02:35:01.425639 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:01.425606 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed76fc49-8f0a-4a64-a273-01cb7a2d612b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv\" (UID: \"ed76fc49-8f0a-4a64-a273-01cb7a2d612b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv" Nov 03 02:35:01.425840 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:01.425763 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnsnx\" (UniqueName: \"kubernetes.io/projected/ed76fc49-8f0a-4a64-a273-01cb7a2d612b-kube-api-access-dnsnx\") pod \"payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv\" (UID: \"ed76fc49-8f0a-4a64-a273-01cb7a2d612b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv" Nov 03 02:35:01.426458 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:01.426439 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed76fc49-8f0a-4a64-a273-01cb7a2d612b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv\" (UID: \"ed76fc49-8f0a-4a64-a273-01cb7a2d612b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv" Nov 03 02:35:01.482843 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:01.482808 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnsnx\" (UniqueName: \"kubernetes.io/projected/ed76fc49-8f0a-4a64-a273-01cb7a2d612b-kube-api-access-dnsnx\") pod \"payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv\" (UID: \"ed76fc49-8f0a-4a64-a273-01cb7a2d612b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv" Nov 03 02:35:01.592684 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:01.592591 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv" Nov 03 02:35:01.983775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:01.983483 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv"] Nov 03 02:35:01.993641 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:35:01.993610 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded76fc49_8f0a_4a64_a273_01cb7a2d612b.slice/crio-a9d0618bcfb9714974310c49214618caf7f7009273dc5e5e634d9fc5dd081e78 WatchSource:0}: Error finding container a9d0618bcfb9714974310c49214618caf7f7009273dc5e5e634d9fc5dd081e78: Status 404 returned error can't find the container with id a9d0618bcfb9714974310c49214618caf7f7009273dc5e5e634d9fc5dd081e78 Nov 03 02:35:02.445745 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:02.445718 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv" Nov 03 02:35:02.445987 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:02.445968 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv" event={"ID":"ed76fc49-8f0a-4a64-a273-01cb7a2d612b","Type":"ContainerStarted","Data":"936a662c6ae831a69413aaeedfa9309906acc3a4bd8b88fefc2a6dbe6436d34e"} Nov 03 02:35:02.446081 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:02.446071 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv" event={"ID":"ed76fc49-8f0a-4a64-a273-01cb7a2d612b","Type":"ContainerStarted","Data":"a9d0618bcfb9714974310c49214618caf7f7009273dc5e5e634d9fc5dd081e78"} Nov 03 02:35:02.572476 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:02.572415 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv" podStartSLOduration=1.572394963 podStartE2EDuration="1.572394963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:02.571455758 +0000 UTC m=+2584.255564915" watchObservedRunningTime="2026-11-03 02:35:02.572394963 +0000 UTC m=+2584.256504119" Nov 03 02:35:04.815735 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:04.815702 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:06.051597 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:06.051560 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87"] Nov 03 02:35:06.099256 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:06.099223 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30d3e735-qqw87"] Nov 03 02:35:06.443652 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:06.443622 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbd12200-ef29-4e84-966d-5faed0aca233" path="/var/lib/kubelet/pods/bbd12200-ef29-4e84-966d-5faed0aca233/volumes" Nov 03 02:35:08.517074 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:08.517043 5463 generic.go:356] "Generic (PLEG): container finished" podID="cec0591b-d4b8-43f5-a031-6ee4e22c96e5" containerID="9597e9b503e48921142c030e540a31efcba4a883220ceb8ee61a4cb3f24081a3" exitCode=0 Nov 03 02:35:08.517628 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:08.517122 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" event={"ID":"cec0591b-d4b8-43f5-a031-6ee4e22c96e5","Type":"ContainerDied","Data":"9597e9b503e48921142c030e540a31efcba4a883220ceb8ee61a4cb3f24081a3"} Nov 03 02:35:08.517628 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:08.517544 5463 scope.go:122] "RemoveContainer" containerID="9597e9b503e48921142c030e540a31efcba4a883220ceb8ee61a4cb3f24081a3" Nov 03 02:35:08.518996 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:08.518970 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:08.519053 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:08.519038 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:08.519091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:08.519071 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:09.526755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:09.526722 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" event={"ID":"cec0591b-d4b8-43f5-a031-6ee4e22c96e5","Type":"ContainerStarted","Data":"c1ea7b9d886cb1f71785ab174a66963c0c78e9c8684d8e61aa20df683d76dc97"} Nov 03 02:35:09.841275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:09.841149 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:10.230837 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.230803 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:35:10.231106 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.231082 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 03 02:35:10.231180 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.231155 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" podUID="cec0591b-d4b8-43f5-a031-6ee4e22c96e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 03 02:35:10.691843 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.691808 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m"] Nov 03 02:35:10.695949 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.695929 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.804608 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.804573 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.804769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.804628 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.804769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.804713 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.804769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.804754 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbsvs\" (UniqueName: \"kubernetes.io/projected/da50d3ef-a1d7-4546-8e52-60ee075543c3-kube-api-access-tbsvs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.804882 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.804818 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.804882 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.804861 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.804960 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.804919 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.805060 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.805031 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da50d3ef-a1d7-4546-8e52-60ee075543c3-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.805156 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.805138 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.828169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.828125 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m"] Nov 03 02:35:10.906911 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.906880 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.907096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.906924 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbsvs\" (UniqueName: \"kubernetes.io/projected/da50d3ef-a1d7-4546-8e52-60ee075543c3-kube-api-access-tbsvs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.907096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.906968 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.907096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.907013 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.907096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.907049 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.907096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.907091 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da50d3ef-a1d7-4546-8e52-60ee075543c3-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.907408 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.907109 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.907408 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.907169 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.907408 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.907203 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.907571 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.907561 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.907742 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.907726 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.908530 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.908503 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.908916 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.908893 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da50d3ef-a1d7-4546-8e52-60ee075543c3-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.909108 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.909082 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.911708 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.910800 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.912345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.912278 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.977351 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.977279 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbsvs\" (UniqueName: \"kubernetes.io/projected/da50d3ef-a1d7-4546-8e52-60ee075543c3-kube-api-access-tbsvs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:10.978352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:10.978314 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:11.009067 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:11.009034 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:11.016082 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:11.016054 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:11.231735 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:11.231638 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 03 02:35:11.231735 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:11.231687 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" podUID="cec0591b-d4b8-43f5-a031-6ee4e22c96e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 03 02:35:11.477456 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:11.477038 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:11.478804 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:35:11.478767 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda50d3ef_a1d7_4546_8e52_60ee075543c3.slice/crio-d09b326ee8c63cebb229a416543f949ada75640ae5434f6dd298e142f5e9858a WatchSource:0}: Error finding container d09b326ee8c63cebb229a416543f949ada75640ae5434f6dd298e142f5e9858a: Status 404 returned error can't find the container with id d09b326ee8c63cebb229a416543f949ada75640ae5434f6dd298e142f5e9858a Nov 03 02:35:11.479712 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:11.479687 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m"] Nov 03 02:35:11.481102 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:11.481064 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:11.481218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:11.481128 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:11.481218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:11.481158 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:11.578219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:11.558588 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" event={"ID":"da50d3ef-a1d7-4546-8e52-60ee075543c3","Type":"ContainerStarted","Data":"d09b326ee8c63cebb229a416543f949ada75640ae5434f6dd298e142f5e9858a"} Nov 03 02:35:11.987547 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:11.987501 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs"] Nov 03 02:35:11.993216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:11.993142 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.098928 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.093976 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-ffba43-openshift-default-dockercfg-vshng\"" Nov 03 02:35:12.148443 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.148311 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:12.155239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.149588 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26968f3-5698-4639-aa81-bedc87536bd6-istio-podinfo\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.155239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.149655 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-workload-socket\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.155239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.149687 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-credential-socket\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.155239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.149760 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-istio-data\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.155239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.149788 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-workload-certs\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.155239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.149840 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26968f3-5698-4639-aa81-bedc87536bd6-istiod-ca-cert\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.155239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.149920 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a26968f3-5698-4639-aa81-bedc87536bd6-istio-token\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.155239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.149994 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtdzk\" (UniqueName: \"kubernetes.io/projected/a26968f3-5698-4639-aa81-bedc87536bd6-kube-api-access-dtdzk\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.155239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.150060 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-istio-envoy\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.207455 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.207421 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs"] Nov 03 02:35:12.241283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.238693 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 03 02:35:12.241283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.238757 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" podUID="cec0591b-d4b8-43f5-a031-6ee4e22c96e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 03 02:35:12.277210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.276603 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26968f3-5698-4639-aa81-bedc87536bd6-istiod-ca-cert\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.277210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.276709 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a26968f3-5698-4639-aa81-bedc87536bd6-istio-token\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.277210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.276935 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtdzk\" (UniqueName: \"kubernetes.io/projected/a26968f3-5698-4639-aa81-bedc87536bd6-kube-api-access-dtdzk\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.277210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.277005 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-istio-envoy\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.277210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.277105 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26968f3-5698-4639-aa81-bedc87536bd6-istio-podinfo\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.308217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.288368 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-workload-socket\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.308217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.288444 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-credential-socket\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.308217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.288585 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-istio-data\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.308217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.288613 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-workload-certs\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.308217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.289302 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-workload-certs\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.308673 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.308574 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26968f3-5698-4639-aa81-bedc87536bd6-istiod-ca-cert\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.312213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.310185 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-istio-data\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.312213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.310438 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-credential-socket\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.312213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.310578 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-workload-socket\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.325658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.325634 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-istio-envoy\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.328254 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.328207 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26968f3-5698-4639-aa81-bedc87536bd6-istio-podinfo\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.406866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.405718 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a26968f3-5698-4639-aa81-bedc87536bd6-istio-token\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.406866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.406151 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtdzk\" (UniqueName: \"kubernetes.io/projected/a26968f3-5698-4639-aa81-bedc87536bd6-kube-api-access-dtdzk\") pod \"e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.552268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.548983 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:12.605370 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.605331 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" event={"ID":"da50d3ef-a1d7-4546-8e52-60ee075543c3","Type":"ContainerStarted","Data":"db84957e8b9cab83d1c05c1b12c15ccc42f959dd0bb276a8bfe3e5ac0dceecf2"} Nov 03 02:35:12.619756 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.619732 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:12.729146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:12.729076 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" podStartSLOduration=2.729058684 podStartE2EDuration="2.729058684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:12.724743323 +0000 UTC m=+2594.408852475" watchObservedRunningTime="2026-11-03 02:35:12.729058684 +0000 UTC m=+2594.413167840" Nov 03 02:35:13.014239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:13.012451 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:13.014239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:13.013646 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 03 02:35:13.014239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:13.013705 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" podUID="da50d3ef-a1d7-4546-8e52-60ee075543c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 03 02:35:13.045125 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:35:13.045092 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:35:13.116345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:13.116234 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs"] Nov 03 02:35:13.117308 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:35:13.117281 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26968f3_5698_4639_aa81_bedc87536bd6.slice/crio-6b6af5741995b4ffd6938ddc3ff4bd8fff80af13fc17e95825896186fc192dff WatchSource:0}: Error finding container 6b6af5741995b4ffd6938ddc3ff4bd8fff80af13fc17e95825896186fc192dff: Status 404 returned error can't find the container with id 6b6af5741995b4ffd6938ddc3ff4bd8fff80af13fc17e95825896186fc192dff Nov 03 02:35:13.120256 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:13.120215 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:13.120442 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:13.120424 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:13.120566 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:13.120553 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:13.232044 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:13.232019 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 03 02:35:13.232153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:13.232082 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" podUID="cec0591b-d4b8-43f5-a031-6ee4e22c96e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 03 02:35:13.617769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:13.617684 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" event={"ID":"a26968f3-5698-4639-aa81-bedc87536bd6","Type":"ContainerStarted","Data":"31489e3b391bc6c54f6718e36e57579f1d901c82194680c6692bde75fdeaeeff"} Nov 03 02:35:13.617769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:13.617743 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" event={"ID":"a26968f3-5698-4639-aa81-bedc87536bd6","Type":"ContainerStarted","Data":"6b6af5741995b4ffd6938ddc3ff4bd8fff80af13fc17e95825896186fc192dff"} Nov 03 02:35:13.621027 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:13.620975 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:13.761215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:13.758025 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" podStartSLOduration=2.758004428 podStartE2EDuration="2.758004428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:13.744529149 +0000 UTC m=+2595.428638295" watchObservedRunningTime="2026-11-03 02:35:13.758004428 +0000 UTC m=+2595.442113585" Nov 03 02:35:14.021626 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:14.021586 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 03 02:35:14.022160 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:14.021653 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" podUID="da50d3ef-a1d7-4546-8e52-60ee075543c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 03 02:35:14.231978 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:14.231939 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 03 02:35:14.232201 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:14.232169 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" podUID="cec0591b-d4b8-43f5-a031-6ee4e22c96e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 03 02:35:14.555218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:14.551979 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv" Nov 03 02:35:14.636094 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:14.636058 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:14.636876 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:14.636848 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:14.637624 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:14.637604 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:35:15.009875 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:15.009832 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 03 02:35:15.010159 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:15.010131 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" podUID="da50d3ef-a1d7-4546-8e52-60ee075543c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 03 02:35:15.277599 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:15.277514 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:35:15.280154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:15.280131 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:35:15.281925 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:15.281909 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:35:15.427107 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:15.427043 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:16.010084 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:16.010049 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 03 02:35:16.010304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:16.010103 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" podUID="da50d3ef-a1d7-4546-8e52-60ee075543c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 03 02:35:16.177214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:16.176569 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:16.625423 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:16.625390 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:17.009693 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.009656 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 03 02:35:17.009909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.009714 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" podUID="da50d3ef-a1d7-4546-8e52-60ee075543c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 03 02:35:17.140572 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.138061 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78"] Nov 03 02:35:17.146223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.145586 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.199563 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.199528 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-957b33-openshift-default-dockercfg-zgcjs\"" Nov 03 02:35:17.219396 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.219367 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:17.281626 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.281527 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78"] Nov 03 02:35:17.287854 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.287174 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-istio-data\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.288030 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.287884 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-workload-certs\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.288030 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.287933 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-istio-envoy\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.288172 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.288117 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96657503-568c-4ac5-a39c-8abd8eadd843-istiod-ca-cert\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.288286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.288262 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-workload-socket\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.288366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.288348 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx7fc\" (UniqueName: \"kubernetes.io/projected/96657503-568c-4ac5-a39c-8abd8eadd843-kube-api-access-mx7fc\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.288536 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.288517 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96657503-568c-4ac5-a39c-8abd8eadd843-istio-token\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.288585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.288578 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96657503-568c-4ac5-a39c-8abd8eadd843-istio-podinfo\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.288661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.288650 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-credential-socket\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.388408 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.388377 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:17.389795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.389769 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96657503-568c-4ac5-a39c-8abd8eadd843-istio-podinfo\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.389943 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.389829 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-credential-socket\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.390032 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.389950 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-istio-data\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.390032 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.390006 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-workload-certs\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.390165 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.390072 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-istio-envoy\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.390165 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.390124 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96657503-568c-4ac5-a39c-8abd8eadd843-istiod-ca-cert\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.390347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.390224 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-workload-socket\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.390347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.390337 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx7fc\" (UniqueName: \"kubernetes.io/projected/96657503-568c-4ac5-a39c-8abd8eadd843-kube-api-access-mx7fc\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.390465 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.390353 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-credential-socket\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.390600 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.390576 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96657503-568c-4ac5-a39c-8abd8eadd843-istio-token\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.390682 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.390658 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-istio-data\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.390935 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.390890 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-workload-socket\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.391036 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.390959 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96657503-568c-4ac5-a39c-8abd8eadd843-istiod-ca-cert\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.391454 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.391417 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-workload-certs\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.393545 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.393519 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-istio-envoy\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.393689 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.393665 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96657503-568c-4ac5-a39c-8abd8eadd843-istio-podinfo\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.486220 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.470846 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96657503-568c-4ac5-a39c-8abd8eadd843-istio-token\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.487103 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.487028 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx7fc\" (UniqueName: \"kubernetes.io/projected/96657503-568c-4ac5-a39c-8abd8eadd843-kube-api-access-mx7fc\") pod \"e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.506750 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.506714 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:17.581846 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.581807 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:17.743264 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:17.742474 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:18.009763 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:18.009729 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 03 02:35:18.009950 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:18.009795 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" podUID="da50d3ef-a1d7-4546-8e52-60ee075543c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 03 02:35:18.107914 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:35:18.107879 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96657503_568c_4ac5_a39c_8abd8eadd843.slice/crio-aed9b2baa11219d85aac6c78fe54a68859177f1340219547895832e6ed9ae347 WatchSource:0}: Error finding container aed9b2baa11219d85aac6c78fe54a68859177f1340219547895832e6ed9ae347: Status 404 returned error can't find the container with id aed9b2baa11219d85aac6c78fe54a68859177f1340219547895832e6ed9ae347 Nov 03 02:35:18.110798 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:18.110411 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78"] Nov 03 02:35:18.111051 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:18.111012 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:18.111129 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:18.111097 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:18.111183 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:18.111135 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:18.723567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:18.723531 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" event={"ID":"96657503-568c-4ac5-a39c-8abd8eadd843","Type":"ContainerStarted","Data":"6241213366f593fb3ddb30fed57fd1ca1218fe0cfe79ad2af59c95ecd85f12fc"} Nov 03 02:35:18.723751 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:18.723625 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" event={"ID":"96657503-568c-4ac5-a39c-8abd8eadd843","Type":"ContainerStarted","Data":"aed9b2baa11219d85aac6c78fe54a68859177f1340219547895832e6ed9ae347"} Nov 03 02:35:18.859836 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:18.859790 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" podStartSLOduration=1.859777112 podStartE2EDuration="1.859777112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:18.858398252 +0000 UTC m=+2600.542507407" watchObservedRunningTime="2026-11-03 02:35:18.859777112 +0000 UTC m=+2600.543886266" Nov 03 02:35:19.042653 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:19.042556 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:19.043526 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:19.043486 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:19.044404 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:19.044381 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:35:19.507933 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:19.507901 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:19.514765 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:19.514738 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:19.732104 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:19.732054 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:19.733298 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:19.733264 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:35:20.742068 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:20.742033 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:20.952080 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:20.951118 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:21.087165 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:21.086906 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw"] Nov 03 02:35:21.107069 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:21.107037 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7883bf4a-xlmtw"] Nov 03 02:35:21.114106 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:21.114066 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:21.922410 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:21.922375 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:22.137782 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:22.137746 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:22.304420 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:22.304347 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:22.445117 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:22.445086 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8ba3dae-180f-4d19-a5d3-fb4dbf253295" path="/var/lib/kubelet/pods/a8ba3dae-180f-4d19-a5d3-fb4dbf253295/volumes" Nov 03 02:35:25.600856 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:25.600819 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:25.815079 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:25.815048 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:26.000848 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:26.000816 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh"] Nov 03 02:35:26.004502 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:26.004483 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh" Nov 03 02:35:26.055793 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:26.055766 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh"] Nov 03 02:35:26.069428 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:26.069354 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trkck\" (UniqueName: \"kubernetes.io/projected/e48e80cd-9263-4d82-8028-2680b548201b-kube-api-access-trkck\") pod \"payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh\" (UID: \"e48e80cd-9263-4d82-8028-2680b548201b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh" Nov 03 02:35:26.069775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:26.069736 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48e80cd-9263-4d82-8028-2680b548201b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh\" (UID: \"e48e80cd-9263-4d82-8028-2680b548201b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh" Nov 03 02:35:26.126212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:26.123229 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-957b33\"" Nov 03 02:35:26.126212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:26.123351 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-957b33-dockercfg-vsvsx\"" Nov 03 02:35:26.176343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:26.173205 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trkck\" (UniqueName: \"kubernetes.io/projected/e48e80cd-9263-4d82-8028-2680b548201b-kube-api-access-trkck\") pod \"payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh\" (UID: \"e48e80cd-9263-4d82-8028-2680b548201b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh" Nov 03 02:35:26.176343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:26.173326 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48e80cd-9263-4d82-8028-2680b548201b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh\" (UID: \"e48e80cd-9263-4d82-8028-2680b548201b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh" Nov 03 02:35:26.176343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:26.174366 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48e80cd-9263-4d82-8028-2680b548201b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh\" (UID: \"e48e80cd-9263-4d82-8028-2680b548201b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh" Nov 03 02:35:26.181580 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:26.181317 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:26.237770 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:26.237735 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trkck\" (UniqueName: \"kubernetes.io/projected/e48e80cd-9263-4d82-8028-2680b548201b-kube-api-access-trkck\") pod \"payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh\" (UID: \"e48e80cd-9263-4d82-8028-2680b548201b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh" Nov 03 02:35:26.318567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:26.318481 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh" Nov 03 02:35:26.693874 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:26.693845 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh"] Nov 03 02:35:26.694517 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:35:26.694489 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48e80cd_9263_4d82_8028_2680b548201b.slice/crio-8fe1f2991a0aaed222b7abed283aa162626989165c1ea0f9e63bf0f6f3074f67 WatchSource:0}: Error finding container 8fe1f2991a0aaed222b7abed283aa162626989165c1ea0f9e63bf0f6f3074f67: Status 404 returned error can't find the container with id 8fe1f2991a0aaed222b7abed283aa162626989165c1ea0f9e63bf0f6f3074f67 Nov 03 02:35:26.828580 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:26.828544 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh" event={"ID":"e48e80cd-9263-4d82-8028-2680b548201b","Type":"ContainerStarted","Data":"e54f8e9f650415c3130ebdd2f559c15c276282461ce7e5d774168bb9f1dc7bd2"} Nov 03 02:35:26.828710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:26.828590 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh" event={"ID":"e48e80cd-9263-4d82-8028-2680b548201b","Type":"ContainerStarted","Data":"8fe1f2991a0aaed222b7abed283aa162626989165c1ea0f9e63bf0f6f3074f67"} Nov 03 02:35:26.828797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:26.828777 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh" Nov 03 02:35:26.946279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:26.946164 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh" podStartSLOduration=1.946150858 podStartE2EDuration="1.946150858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:26.942610652 +0000 UTC m=+2608.626719808" watchObservedRunningTime="2026-11-03 02:35:26.946150858 +0000 UTC m=+2608.630260012" Nov 03 02:35:31.536858 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:31.536817 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:36.321719 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:36.321687 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:37.840557 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:37.840523 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh" Nov 03 02:35:40.967500 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:40.967417 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:41.129362 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:41.129326 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:41.266263 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:41.266167 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:43.012133 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:35:43.012091 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:35:45.869574 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:45.869541 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:46.332594 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:46.332560 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:47.136219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.136171 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:47.618125 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.618091 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8"] Nov 03 02:35:47.622319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.622293 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.706218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.705678 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-4616ec-openshift-default-dockercfg-kkpxk\"" Nov 03 02:35:47.731210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.727788 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-data\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.731210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.727848 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-envoy\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.731210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.727877 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-workload-certs\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.731210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.727923 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-podinfo\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.731210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.727976 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-workload-socket\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.731210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.728030 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t74r2\" (UniqueName: \"kubernetes.io/projected/b64037fe-08de-4969-bdc0-97a3802aa28c-kube-api-access-t74r2\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.731210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.728061 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-credential-socket\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.757212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.753416 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-token\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.757212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.753465 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64037fe-08de-4969-bdc0-97a3802aa28c-istiod-ca-cert\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.831548 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.831503 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:47.884149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.881701 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8"] Nov 03 02:35:47.938203 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.938148 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-workload-socket\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.938684 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.938658 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t74r2\" (UniqueName: \"kubernetes.io/projected/b64037fe-08de-4969-bdc0-97a3802aa28c-kube-api-access-t74r2\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.939238 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.938816 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-credential-socket\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.939494 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.939474 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-token\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.939864 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.939844 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64037fe-08de-4969-bdc0-97a3802aa28c-istiod-ca-cert\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.940427 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.940120 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-data\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.940582 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.940568 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-envoy\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.940958 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.940814 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-credential-socket\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.941038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.940933 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-workload-certs\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.941167 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.941050 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-podinfo\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.941992 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.941969 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-workload-certs\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.942821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.942505 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-data\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.943589 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.943562 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-workload-socket\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.944237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.943984 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64037fe-08de-4969-bdc0-97a3802aa28c-istiod-ca-cert\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.945655 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.945632 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-envoy\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.951151 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.951129 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-podinfo\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:47.998684 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:47.998651 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-token\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:48.002016 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:48.001992 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t74r2\" (UniqueName: \"kubernetes.io/projected/b64037fe-08de-4969-bdc0-97a3802aa28c-kube-api-access-t74r2\") pod \"e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:48.159633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:48.159597 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:48.237204 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:48.237149 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:48.657733 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:35:48.657668 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64037fe_08de_4969_bdc0_97a3802aa28c.slice/crio-025675925ce4ac652b63676d496b9bdd1331d82d4037810b89dc8aa598e9d723 WatchSource:0}: Error finding container 025675925ce4ac652b63676d496b9bdd1331d82d4037810b89dc8aa598e9d723: Status 404 returned error can't find the container with id 025675925ce4ac652b63676d496b9bdd1331d82d4037810b89dc8aa598e9d723 Nov 03 02:35:48.660751 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:48.660710 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:48.660869 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:48.660801 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:48.660869 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:48.660849 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:35:48.661721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:48.661698 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8"] Nov 03 02:35:49.020758 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:49.020658 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" event={"ID":"b64037fe-08de-4969-bdc0-97a3802aa28c","Type":"ContainerStarted","Data":"c2c1e3bff8da2b71db8a98253ede81008c60ec15e7607e7047dc2ee9edc3d37b"} Nov 03 02:35:49.020758 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:49.020713 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" event={"ID":"b64037fe-08de-4969-bdc0-97a3802aa28c","Type":"ContainerStarted","Data":"025675925ce4ac652b63676d496b9bdd1331d82d4037810b89dc8aa598e9d723"} Nov 03 02:35:49.139236 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:49.139176 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" podStartSLOduration=2.139162772 podStartE2EDuration="2.139162772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:49.136549946 +0000 UTC m=+2630.820659124" watchObservedRunningTime="2026-11-03 02:35:49.139162772 +0000 UTC m=+2630.823271926" Nov 03 02:35:49.237882 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:49.237849 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:49.244463 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:49.244433 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:50.028153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:50.028121 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:50.029034 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:50.029009 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:35:51.534363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:51.534319 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:52.706675 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:52.706641 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:52.873701 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:52.873664 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:55.322295 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:55.322265 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:55.840716 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:55.840671 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:56.023524 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:56.023486 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:56.770301 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:35:56.770258 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:01.488068 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:01.488029 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:01.664935 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:01.664877 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:01.864850 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:01.864753 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:04.912066 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:04.912035 5463 scope.go:122] "RemoveContainer" containerID="a261ce48fda2bc6384dadf9f6f99837552287e4c76eaea1948ab59c4381e6dee" Nov 03 02:36:04.924169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:04.924149 5463 scope.go:122] "RemoveContainer" containerID="30ac2cc6b30c8a6bff14c79e3162945128e60f455a45f7c78e1aba6d745d8406" Nov 03 02:36:06.393987 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:06.393939 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:09.190668 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:09.190635 5463 generic.go:356] "Generic (PLEG): container finished" podID="b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" containerID="7f7d2be25db222b2001948ef40d630d02d2bd4dfb47b2693fad9596345175625" exitCode=0 Nov 03 02:36:09.191120 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:09.190715 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" event={"ID":"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a","Type":"ContainerDied","Data":"7f7d2be25db222b2001948ef40d630d02d2bd4dfb47b2693fad9596345175625"} Nov 03 02:36:09.191274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:09.191255 5463 scope.go:122] "RemoveContainer" containerID="7f7d2be25db222b2001948ef40d630d02d2bd4dfb47b2693fad9596345175625" Nov 03 02:36:09.193102 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:09.193073 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:36:09.193175 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:09.193146 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:36:09.193247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:09.193179 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:36:09.973695 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:09.970278 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:36:10.202516 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:10.202475 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" event={"ID":"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a","Type":"ContainerStarted","Data":"b73a5b35a90e2f33a5ad66801f7ffc9a1ec78093c2b50c18c6855e40048d936a"} Nov 03 02:36:10.514173 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:10.514091 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:36:10.516003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:10.515976 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 03 02:36:10.516304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:10.516225 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" podUID="b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 03 02:36:11.221306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:11.221274 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:11.514292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:11.514211 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 03 02:36:11.514292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:11.514278 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" podUID="b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 03 02:36:12.514491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:12.514457 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 03 02:36:12.514886 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:12.514519 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" podUID="b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 03 02:36:13.013643 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:36:13.013611 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:36:13.514132 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:13.514101 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 03 02:36:13.514359 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:13.514152 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" podUID="b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 03 02:36:14.514163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:14.514125 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 03 02:36:14.514581 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:14.514185 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" podUID="b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 03 02:36:15.519938 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:15.519910 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:36:15.520412 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:15.520279 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:36:15.520993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:15.520972 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 02:36:15.927366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:15.927330 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:17.911337 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:17.911296 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:20.568652 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:20.568612 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:22.161058 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:22.161025 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:23.629861 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:23.629821 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:23.780284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:23.780248 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:25.513757 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:25.513722 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:30.171320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:30.171283 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:30.314444 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:30.314403 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:30.612007 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:30.611921 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:35.174185 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:35.174094 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:37.463563 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:37.463530 5463 generic.go:356] "Generic (PLEG): container finished" podID="8f8b0d5f-94b6-439a-bbb0-35fb828131dc" containerID="fbaeb2432e1d7a5bf0a9433268d8a332624e3c7d5158bfbb054a60d28513baab" exitCode=0 Nov 03 02:36:37.464009 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:37.463603 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" event={"ID":"8f8b0d5f-94b6-439a-bbb0-35fb828131dc","Type":"ContainerDied","Data":"fbaeb2432e1d7a5bf0a9433268d8a332624e3c7d5158bfbb054a60d28513baab"} Nov 03 02:36:37.464055 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:37.464025 5463 scope.go:122] "RemoveContainer" containerID="fbaeb2432e1d7a5bf0a9433268d8a332624e3c7d5158bfbb054a60d28513baab" Nov 03 02:36:37.465715 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:37.465682 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:36:37.465842 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:37.465773 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:36:37.465842 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:37.465825 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:36:38.499279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:38.499238 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" event={"ID":"8f8b0d5f-94b6-439a-bbb0-35fb828131dc","Type":"ContainerStarted","Data":"f03db564fe7b11373e7598a551d7d2fc033384c465a12387536d243ba12b8082"} Nov 03 02:36:39.460394 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:39.460359 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:36:39.460843 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:39.460821 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Nov 03 02:36:39.460998 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:39.460974 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" podUID="8f8b0d5f-94b6-439a-bbb0-35fb828131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Nov 03 02:36:39.866179 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:39.866095 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:40.339475 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:40.339438 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:36:40.460725 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:40.460693 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Nov 03 02:36:40.461014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:40.460985 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" podUID="8f8b0d5f-94b6-439a-bbb0-35fb828131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Nov 03 02:36:41.461623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:41.461586 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Nov 03 02:36:41.462248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:41.462214 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" podUID="8f8b0d5f-94b6-439a-bbb0-35fb828131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Nov 03 02:36:42.460212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:42.459555 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Nov 03 02:36:42.460212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:42.459626 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" podUID="8f8b0d5f-94b6-439a-bbb0-35fb828131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Nov 03 02:36:43.012470 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:36:43.012432 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:36:43.455685 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:43.455651 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Nov 03 02:36:43.455897 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:43.455715 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" podUID="8f8b0d5f-94b6-439a-bbb0-35fb828131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Nov 03 02:36:44.461283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:44.461253 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:36:44.462376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:44.462350 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 02:36:44.537020 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:44.536966 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:49.338806 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:49.338755 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:54.120800 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:54.120766 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:58.926673 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:58.926646 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:59.638585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:59.638513 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Nov 03 02:36:59.638585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:59.638567 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" podUID="a26968f3-5698-4639-aa81-bedc87536bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Nov 03 02:36:59.711835 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:59.711803 5463 generic.go:356] "Generic (PLEG): container finished" podID="96657503-568c-4ac5-a39c-8abd8eadd843" containerID="6241213366f593fb3ddb30fed57fd1ca1218fe0cfe79ad2af59c95ecd85f12fc" exitCode=0 Nov 03 02:36:59.711984 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:59.711877 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" event={"ID":"96657503-568c-4ac5-a39c-8abd8eadd843","Type":"ContainerDied","Data":"6241213366f593fb3ddb30fed57fd1ca1218fe0cfe79ad2af59c95ecd85f12fc"} Nov 03 02:36:59.712388 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:59.712367 5463 scope.go:122] "RemoveContainer" containerID="6241213366f593fb3ddb30fed57fd1ca1218fe0cfe79ad2af59c95ecd85f12fc" Nov 03 02:36:59.713663 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:59.713644 5463 generic.go:356] "Generic (PLEG): container finished" podID="a26968f3-5698-4639-aa81-bedc87536bd6" containerID="31489e3b391bc6c54f6718e36e57579f1d901c82194680c6692bde75fdeaeeff" exitCode=0 Nov 03 02:36:59.713794 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:59.713689 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" event={"ID":"a26968f3-5698-4639-aa81-bedc87536bd6","Type":"ContainerDied","Data":"31489e3b391bc6c54f6718e36e57579f1d901c82194680c6692bde75fdeaeeff"} Nov 03 02:36:59.714089 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:59.714050 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:36:59.714089 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:59.714071 5463 scope.go:122] "RemoveContainer" containerID="31489e3b391bc6c54f6718e36e57579f1d901c82194680c6692bde75fdeaeeff" Nov 03 02:36:59.714268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:59.714121 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:36:59.714268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:59.714150 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:36:59.715620 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:59.715594 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:36:59.715692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:59.715655 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:36:59.715692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:36:59.715685 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:37:00.010087 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:00.009579 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:00.728233 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:00.726390 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" event={"ID":"96657503-568c-4ac5-a39c-8abd8eadd843","Type":"ContainerStarted","Data":"831eb67657d07363ad876d9bcccb3dfb16fcc0e8a7e15edc080fccb64f59212c"} Nov 03 02:37:00.729542 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:00.729506 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" event={"ID":"a26968f3-5698-4639-aa81-bedc87536bd6","Type":"ContainerStarted","Data":"ed88cf210f0fde048e7a0868854a4e698095edad0dc9e498075a5e8953a9cb48"} Nov 03 02:37:01.507890 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:01.507852 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:37:01.509345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:01.509314 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 03 02:37:01.509496 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:01.509383 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" podUID="96657503-568c-4ac5-a39c-8abd8eadd843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 03 02:37:01.620616 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:01.620578 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:37:01.620799 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:01.620719 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Nov 03 02:37:01.620799 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:01.620764 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" podUID="a26968f3-5698-4639-aa81-bedc87536bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Nov 03 02:37:02.508530 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:02.508499 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 03 02:37:02.508950 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:02.508563 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" podUID="96657503-568c-4ac5-a39c-8abd8eadd843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 03 02:37:02.620821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:02.620789 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Nov 03 02:37:02.620999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:02.620840 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" podUID="a26968f3-5698-4639-aa81-bedc87536bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Nov 03 02:37:02.911593 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:02.910561 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:37:02.911770 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:02.911597 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:37:03.507624 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:03.507589 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 03 02:37:03.507872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:03.507657 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" podUID="96657503-568c-4ac5-a39c-8abd8eadd843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 03 02:37:03.621086 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:03.621056 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Nov 03 02:37:03.621488 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:03.621118 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" podUID="a26968f3-5698-4639-aa81-bedc87536bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Nov 03 02:37:03.897879 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:03.897844 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:04.072392 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:04.072325 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:04.219457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:04.219335 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:04.508158 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:04.508075 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 03 02:37:04.508158 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:04.508137 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" podUID="96657503-568c-4ac5-a39c-8abd8eadd843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 03 02:37:04.620880 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:04.620848 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Nov 03 02:37:04.621063 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:04.620911 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" podUID="a26968f3-5698-4639-aa81-bedc87536bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Nov 03 02:37:04.732003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:04.731966 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:37:04.967218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:04.964392 5463 scope.go:122] "RemoveContainer" containerID="dc7a4fa162e005dcf75bb8856db2c837e987e0ff24be5744522e8187a5083981" Nov 03 02:37:05.341725 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:05.341653 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:05.507820 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:05.507733 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 03 02:37:05.507820 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:05.507784 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" podUID="96657503-568c-4ac5-a39c-8abd8eadd843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 03 02:37:05.620965 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:05.620882 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Nov 03 02:37:05.620965 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:05.620944 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" podUID="a26968f3-5698-4639-aa81-bedc87536bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Nov 03 02:37:06.512721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:06.512688 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:37:06.513793 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:06.513772 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 02:37:06.626555 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:06.626529 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:37:06.626883 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:06.626856 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:37:06.639461 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:06.639434 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 02:37:08.766297 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:08.766264 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:13.009208 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:37:13.009160 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:37:13.439306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:13.439268 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:18.301489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:18.301446 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:18.449221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:18.449161 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:23.117109 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:23.117044 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:25.823683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:25.823645 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:28.127263 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:28.127185 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:32.148321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:32.148287 5463 generic.go:356] "Generic (PLEG): container finished" podID="b64037fe-08de-4969-bdc0-97a3802aa28c" containerID="c2c1e3bff8da2b71db8a98253ede81008c60ec15e7607e7047dc2ee9edc3d37b" exitCode=0 Nov 03 02:37:32.148784 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:32.148448 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" event={"ID":"b64037fe-08de-4969-bdc0-97a3802aa28c","Type":"ContainerDied","Data":"c2c1e3bff8da2b71db8a98253ede81008c60ec15e7607e7047dc2ee9edc3d37b"} Nov 03 02:37:32.148965 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:32.148945 5463 scope.go:122] "RemoveContainer" containerID="c2c1e3bff8da2b71db8a98253ede81008c60ec15e7607e7047dc2ee9edc3d37b" Nov 03 02:37:32.151167 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:32.151133 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:37:32.151350 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:32.151313 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:37:32.151403 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:32.151388 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:37:32.798167 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:32.798127 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:33.182421 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:33.182371 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" event={"ID":"b64037fe-08de-4969-bdc0-97a3802aa28c","Type":"ContainerStarted","Data":"574f6ae076f08335342363ef712fc820155bfd1b14bb511aae5faead6570bdb9"} Nov 03 02:37:33.237703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:33.237670 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:37:33.239108 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:33.239085 5463 patch_prober.go:29] interesting pod/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body= Nov 03 02:37:33.239167 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:33.239136 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" podUID="b64037fe-08de-4969-bdc0-97a3802aa28c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" Nov 03 02:37:34.238278 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:34.238247 5463 patch_prober.go:29] interesting pod/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body= Nov 03 02:37:34.238664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:34.238309 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" podUID="b64037fe-08de-4969-bdc0-97a3802aa28c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" Nov 03 02:37:35.028053 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:35.028023 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:37:35.237595 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:35.237561 5463 patch_prober.go:29] interesting pod/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body= Nov 03 02:37:35.237768 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:35.237610 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" podUID="b64037fe-08de-4969-bdc0-97a3802aa28c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" Nov 03 02:37:36.237902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:36.237865 5463 patch_prober.go:29] interesting pod/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body= Nov 03 02:37:36.238436 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:36.237921 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" podUID="b64037fe-08de-4969-bdc0-97a3802aa28c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" Nov 03 02:37:37.237577 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:37.237543 5463 patch_prober.go:29] interesting pod/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body= Nov 03 02:37:37.237773 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:37.237606 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" podUID="b64037fe-08de-4969-bdc0-97a3802aa28c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" Nov 03 02:37:37.487322 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:37.487275 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:38.238343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:38.238308 5463 patch_prober.go:29] interesting pod/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body= Nov 03 02:37:38.238541 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:38.238371 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" podUID="b64037fe-08de-4969-bdc0-97a3802aa28c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" Nov 03 02:37:39.242909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:39.242878 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:37:39.243952 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:39.243932 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:37:42.290520 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:42.276775 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:43.010597 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:37:43.010562 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:37:45.177869 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:45.177831 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:47.479849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:47.479810 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:52.344308 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:52.344273 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:55.252293 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:55.252260 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:57.326484 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:37:57.326424 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:02.031166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:02.031130 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:06.702494 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:06.702456 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:07.018025 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.017945 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k"] Nov 03 02:38:07.021410 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.021389 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:07.077068 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.077037 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k"] Nov 03 02:38:07.079029 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.078999 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5effe46b-a96c-46ab-ae45-188c31537fbd-tls-certs\") pod \"auth-test-model-3a4683-kserve-84bc685497-rgm4k\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:07.079184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.079051 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-home\") pod \"auth-test-model-3a4683-kserve-84bc685497-rgm4k\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:07.079184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.079157 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-tmp-dir\") pod \"auth-test-model-3a4683-kserve-84bc685497-rgm4k\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:07.079184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.079181 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-model-cache\") pod \"auth-test-model-3a4683-kserve-84bc685497-rgm4k\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:07.079326 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.079313 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q8bl\" (UniqueName: \"kubernetes.io/projected/5effe46b-a96c-46ab-ae45-188c31537fbd-kube-api-access-6q8bl\") pod \"auth-test-model-3a4683-kserve-84bc685497-rgm4k\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:07.079379 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.079366 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-dshm\") pod \"auth-test-model-3a4683-kserve-84bc685497-rgm4k\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:07.122686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.122657 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-3a4683\"/\"openshift-service-ca.crt\"" Nov 03 02:38:07.122686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.122666 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-3a4683\"/\"default-dockercfg-9qvwg\"" Nov 03 02:38:07.122868 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.122724 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-3a4683\"/\"auth-test-model-3a4683-kserve-self-signed-certs\"" Nov 03 02:38:07.124465 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.124438 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-3a4683\"/\"kube-root-ca.crt\"" Nov 03 02:38:07.131040 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.131018 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:07.181805 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.181772 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5effe46b-a96c-46ab-ae45-188c31537fbd-tls-certs\") pod \"auth-test-model-3a4683-kserve-84bc685497-rgm4k\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:07.182059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.181832 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-home\") pod \"auth-test-model-3a4683-kserve-84bc685497-rgm4k\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:07.182059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.181906 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-tmp-dir\") pod \"auth-test-model-3a4683-kserve-84bc685497-rgm4k\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:07.182059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.181930 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-model-cache\") pod \"auth-test-model-3a4683-kserve-84bc685497-rgm4k\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:07.182059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.182022 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q8bl\" (UniqueName: \"kubernetes.io/projected/5effe46b-a96c-46ab-ae45-188c31537fbd-kube-api-access-6q8bl\") pod \"auth-test-model-3a4683-kserve-84bc685497-rgm4k\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:07.182437 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.182104 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-dshm\") pod \"auth-test-model-3a4683-kserve-84bc685497-rgm4k\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:07.182437 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.182316 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-home\") pod \"auth-test-model-3a4683-kserve-84bc685497-rgm4k\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:07.182555 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.182493 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-tmp-dir\") pod \"auth-test-model-3a4683-kserve-84bc685497-rgm4k\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:07.182685 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.182647 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-model-cache\") pod \"auth-test-model-3a4683-kserve-84bc685497-rgm4k\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:07.185092 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.185067 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-dshm\") pod \"auth-test-model-3a4683-kserve-84bc685497-rgm4k\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:07.185927 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.185903 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5effe46b-a96c-46ab-ae45-188c31537fbd-tls-certs\") pod \"auth-test-model-3a4683-kserve-84bc685497-rgm4k\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:07.239721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.239693 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q8bl\" (UniqueName: \"kubernetes.io/projected/5effe46b-a96c-46ab-ae45-188c31537fbd-kube-api-access-6q8bl\") pod \"auth-test-model-3a4683-kserve-84bc685497-rgm4k\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:07.333567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.333486 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:07.704524 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:38:07.704482 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5effe46b_a96c_46ab_ae45_188c31537fbd.slice/crio-9c8ce3f44f2220a452e1310853918a11e6aef8cc0629085a46d6b5e5f04a8e41 WatchSource:0}: Error finding container 9c8ce3f44f2220a452e1310853918a11e6aef8cc0629085a46d6b5e5f04a8e41: Status 404 returned error can't find the container with id 9c8ce3f44f2220a452e1310853918a11e6aef8cc0629085a46d6b5e5f04a8e41 Nov 03 02:38:07.708268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:07.708247 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k"] Nov 03 02:38:08.478146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:08.478112 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" event={"ID":"5effe46b-a96c-46ab-ae45-188c31537fbd","Type":"ContainerStarted","Data":"b03ba6a91609d9a0cafc23dcff960ee1ceac37e14db76888a5cc228103496409"} Nov 03 02:38:08.478380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:08.478151 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" event={"ID":"5effe46b-a96c-46ab-ae45-188c31537fbd","Type":"ContainerStarted","Data":"9c8ce3f44f2220a452e1310853918a11e6aef8cc0629085a46d6b5e5f04a8e41"} Nov 03 02:38:08.597421 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:08.597369 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" podStartSLOduration=2.597356484 podStartE2EDuration="2.597356484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:38:08.59347548 +0000 UTC m=+2770.277584649" watchObservedRunningTime="2026-11-03 02:38:08.597356484 +0000 UTC m=+2770.281465665" Nov 03 02:38:09.333768 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:09.333735 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:11.414020 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:11.413983 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:13.010045 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:38:13.010011 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:38:16.161501 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:16.161462 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:17.099094 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:17.099036 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr"] Nov 03 02:38:17.102564 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:17.102540 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a3eeb8d-4qsbr"] Nov 03 02:38:17.333607 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:17.333580 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:17.346339 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:17.346314 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:17.561439 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:17.561408 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:38:18.446918 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:18.446886 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8e096d0-d67b-452a-bd08-870c210c52c5" path="/var/lib/kubelet/pods/c8e096d0-d67b-452a-bd08-870c210c52c5/volumes" Nov 03 02:38:18.542584 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:18.542553 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:20.931552 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:20.931515 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:25.831954 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:25.831918 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:27.819231 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:27.819169 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:29.238735 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:29.238706 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn"] Nov 03 02:38:29.239174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:29.238899 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" podUID="4d0ba391-497c-4575-909a-10a65432678d" containerName="istio-proxy" containerID="cri-o://de5df43669c4345c1b55cadf63cf6a5283762a5c14a1ab83aee295a947c2ed06" gracePeriod=30 Nov 03 02:38:30.785903 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:30.785863 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:30.964466 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:30.964429 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:34.523202 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.523126 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:38:34.616833 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.616805 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-istio-envoy\") pod \"4d0ba391-497c-4575-909a-10a65432678d\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " Nov 03 02:38:34.617022 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.616840 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0ba391-497c-4575-909a-10a65432678d-istio-token\" (UniqueName: \"kubernetes.io/projected/4d0ba391-497c-4575-909a-10a65432678d-istio-token\") pod \"4d0ba391-497c-4575-909a-10a65432678d\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " Nov 03 02:38:34.617022 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.616904 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-istio-data\") pod \"4d0ba391-497c-4575-909a-10a65432678d\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " Nov 03 02:38:34.617022 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.616939 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-workload-certs\") pod \"4d0ba391-497c-4575-909a-10a65432678d\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " Nov 03 02:38:34.617221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.617030 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-credential-socket\") pod \"4d0ba391-497c-4575-909a-10a65432678d\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " Nov 03 02:38:34.617221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.617091 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-workload-socket\") pod \"4d0ba391-497c-4575-909a-10a65432678d\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " Nov 03 02:38:34.617221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.617159 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d0ba391-497c-4575-909a-10a65432678d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0ba391-497c-4575-909a-10a65432678d-istiod-ca-cert\") pod \"4d0ba391-497c-4575-909a-10a65432678d\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " Nov 03 02:38:34.617405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.617263 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-istio-data" pod "4d0ba391-497c-4575-909a-10a65432678d" (UID: "4d0ba391-497c-4575-909a-10a65432678d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:34.617405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.617275 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0ba391-497c-4575-909a-10a65432678d-kube-api-access-7m9zl\" (UniqueName: \"kubernetes.io/projected/4d0ba391-497c-4575-909a-10a65432678d-kube-api-access-7m9zl\") pod \"4d0ba391-497c-4575-909a-10a65432678d\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " Nov 03 02:38:34.617405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.617287 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-workload-certs" pod "4d0ba391-497c-4575-909a-10a65432678d" (UID: "4d0ba391-497c-4575-909a-10a65432678d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:34.617405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.617324 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d0ba391-497c-4575-909a-10a65432678d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0ba391-497c-4575-909a-10a65432678d-istio-podinfo\") pod \"4d0ba391-497c-4575-909a-10a65432678d\" (UID: \"4d0ba391-497c-4575-909a-10a65432678d\") " Nov 03 02:38:34.617687 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.617655 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0ba391-497c-4575-909a-10a65432678d-istiod-ca-cert" pod "4d0ba391-497c-4575-909a-10a65432678d" (UID: "4d0ba391-497c-4575-909a-10a65432678d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:38:34.618142 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.618113 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0ba391-497c-4575-909a-10a65432678d-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:34.618272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.618144 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:34.618272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.618159 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:34.618470 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.618417 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-workload-socket" pod "4d0ba391-497c-4575-909a-10a65432678d" (UID: "4d0ba391-497c-4575-909a-10a65432678d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:34.618470 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.618456 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-credential-socket" pod "4d0ba391-497c-4575-909a-10a65432678d" (UID: "4d0ba391-497c-4575-909a-10a65432678d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:34.619382 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.619346 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-istio-envoy" pod "4d0ba391-497c-4575-909a-10a65432678d" (UID: "4d0ba391-497c-4575-909a-10a65432678d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:34.619566 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.619542 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0ba391-497c-4575-909a-10a65432678d-istio-token" pod "4d0ba391-497c-4575-909a-10a65432678d" (UID: "4d0ba391-497c-4575-909a-10a65432678d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:34.619749 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.619724 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d0ba391-497c-4575-909a-10a65432678d-istio-podinfo" pod "4d0ba391-497c-4575-909a-10a65432678d" (UID: "4d0ba391-497c-4575-909a-10a65432678d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:38:34.619875 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.619856 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0ba391-497c-4575-909a-10a65432678d-kube-api-access-7m9zl" pod "4d0ba391-497c-4575-909a-10a65432678d" (UID: "4d0ba391-497c-4575-909a-10a65432678d"). InnerVolumeSpecName "kube-api-access-7m9zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:34.719544 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.719503 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:34.719544 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.719543 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d0ba391-497c-4575-909a-10a65432678d-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:34.719814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.719559 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:34.719814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.719574 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0ba391-497c-4575-909a-10a65432678d-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:34.719814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.719589 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m9zl\" (UniqueName: \"kubernetes.io/projected/4d0ba391-497c-4575-909a-10a65432678d-kube-api-access-7m9zl\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:34.719814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.719603 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0ba391-497c-4575-909a-10a65432678d-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:34.729130 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.729093 5463 generic.go:356] "Generic (PLEG): container finished" podID="4d0ba391-497c-4575-909a-10a65432678d" containerID="de5df43669c4345c1b55cadf63cf6a5283762a5c14a1ab83aee295a947c2ed06" exitCode=0 Nov 03 02:38:34.729325 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.729140 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" Nov 03 02:38:34.729325 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.729173 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" event={"ID":"4d0ba391-497c-4575-909a-10a65432678d","Type":"ContainerDied","Data":"de5df43669c4345c1b55cadf63cf6a5283762a5c14a1ab83aee295a947c2ed06"} Nov 03 02:38:34.729325 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.729298 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn" event={"ID":"4d0ba391-497c-4575-909a-10a65432678d","Type":"ContainerDied","Data":"a586a3387e3ea79d6a20fd52380d6308ad52e21c5e5addfe53c77dff691ce3a1"} Nov 03 02:38:34.729498 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.729336 5463 scope.go:122] "RemoveContainer" containerID="de5df43669c4345c1b55cadf63cf6a5283762a5c14a1ab83aee295a947c2ed06" Nov 03 02:38:34.745267 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.745209 5463 scope.go:122] "RemoveContainer" containerID="de5df43669c4345c1b55cadf63cf6a5283762a5c14a1ab83aee295a947c2ed06" Nov 03 02:38:34.745533 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:38:34.745513 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5df43669c4345c1b55cadf63cf6a5283762a5c14a1ab83aee295a947c2ed06\": container with ID starting with de5df43669c4345c1b55cadf63cf6a5283762a5c14a1ab83aee295a947c2ed06 not found: ID does not exist" containerID="de5df43669c4345c1b55cadf63cf6a5283762a5c14a1ab83aee295a947c2ed06" Nov 03 02:38:34.745590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.745542 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5df43669c4345c1b55cadf63cf6a5283762a5c14a1ab83aee295a947c2ed06"} err="failed to get container status \"de5df43669c4345c1b55cadf63cf6a5283762a5c14a1ab83aee295a947c2ed06\": rpc error: code = NotFound desc = could not find container \"de5df43669c4345c1b55cadf63cf6a5283762a5c14a1ab83aee295a947c2ed06\": container with ID starting with de5df43669c4345c1b55cadf63cf6a5283762a5c14a1ab83aee295a947c2ed06 not found: ID does not exist" Nov 03 02:38:34.896578 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.896549 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn"] Nov 03 02:38:34.901800 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:34.901776 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7wwkn"] Nov 03 02:38:35.728562 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:35.728509 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:36.444906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:36.444864 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d0ba391-497c-4575-909a-10a65432678d" path="/var/lib/kubelet/pods/4d0ba391-497c-4575-909a-10a65432678d/volumes" Nov 03 02:38:40.437931 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:40.437849 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:43.099214 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:38:43.087503 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:38:45.209469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:45.209426 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:49.951892 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:49.951856 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:54.376815 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:54.376753 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:54.590878 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:54.590845 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:59.262325 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:59.262274 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:59.462434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:59.462390 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf"] Nov 03 02:38:59.463059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:38:59.462746 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" podUID="a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" containerName="istio-proxy" containerID="cri-o://0354686078f514d804a4f824641bf6f01d75020b6bf11b89a176ddc768bbcf35" gracePeriod=30 Nov 03 02:39:02.217314 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:02.217272 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:02.735382 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:02.735346 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:02.883653 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:02.883619 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:04.052330 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.052269 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:04.745940 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.745885 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:39:04.867477 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.867444 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-workload-socket\") pod \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " Nov 03 02:39:04.867683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.867498 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-envoy\") pod \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " Nov 03 02:39:04.867683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.867592 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-token\" (UniqueName: \"kubernetes.io/projected/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-token\") pod \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " Nov 03 02:39:04.867683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.867632 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-kube-api-access-wg89m\" (UniqueName: \"kubernetes.io/projected/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-kube-api-access-wg89m\") pod \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " Nov 03 02:39:04.867875 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.867729 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istiod-ca-cert\") pod \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " Nov 03 02:39:04.867875 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.867765 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-credential-socket\") pod \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " Nov 03 02:39:04.867875 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.867779 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-workload-socket" pod "a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" (UID: "a9db59ca-ab6b-4c1d-b24c-bcea3efc4268"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:04.867875 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.867820 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-data\") pod \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " Nov 03 02:39:04.868203 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.868151 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-data" pod "a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" (UID: "a9db59ca-ab6b-4c1d-b24c-bcea3efc4268"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:04.868343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.868232 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-workload-certs\") pod \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " Nov 03 02:39:04.868343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.868281 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-podinfo\") pod \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\" (UID: \"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268\") " Nov 03 02:39:04.868343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.868282 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istiod-ca-cert" pod "a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" (UID: "a9db59ca-ab6b-4c1d-b24c-bcea3efc4268"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:39:04.868565 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.868541 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-workload-certs" pod "a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" (UID: "a9db59ca-ab6b-4c1d-b24c-bcea3efc4268"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:04.869091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.868996 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:04.869091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.869023 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:04.869091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.869038 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:04.869091 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.869051 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:04.869681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.869638 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-credential-socket" pod "a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" (UID: "a9db59ca-ab6b-4c1d-b24c-bcea3efc4268"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:04.871030 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.870980 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-podinfo" pod "a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" (UID: "a9db59ca-ab6b-4c1d-b24c-bcea3efc4268"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:39:04.871030 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.870983 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-kube-api-access-wg89m" pod "a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" (UID: "a9db59ca-ab6b-4c1d-b24c-bcea3efc4268"). InnerVolumeSpecName "kube-api-access-wg89m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:04.871030 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.871019 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-envoy" pod "a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" (UID: "a9db59ca-ab6b-4c1d-b24c-bcea3efc4268"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:04.871276 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.871183 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-token" pod "a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" (UID: "a9db59ca-ab6b-4c1d-b24c-bcea3efc4268"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:04.970566 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.970521 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:04.970566 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.970562 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:04.970825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.970580 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg89m\" (UniqueName: \"kubernetes.io/projected/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-kube-api-access-wg89m\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:04.970825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.970596 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:04.970825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:04.970609 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:05.024859 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:05.024739 5463 generic.go:356] "Generic (PLEG): container finished" podID="a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" containerID="0354686078f514d804a4f824641bf6f01d75020b6bf11b89a176ddc768bbcf35" exitCode=0 Nov 03 02:39:05.024859 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:05.024800 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" event={"ID":"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268","Type":"ContainerDied","Data":"0354686078f514d804a4f824641bf6f01d75020b6bf11b89a176ddc768bbcf35"} Nov 03 02:39:05.024859 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:05.024811 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" Nov 03 02:39:05.024859 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:05.024851 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf" event={"ID":"a9db59ca-ab6b-4c1d-b24c-bcea3efc4268","Type":"ContainerDied","Data":"7b8b3a16316e4eb1c76dddd245bab36eb361610689ff003daf34bac066bfac75"} Nov 03 02:39:05.026392 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:05.024890 5463 scope.go:122] "RemoveContainer" containerID="0354686078f514d804a4f824641bf6f01d75020b6bf11b89a176ddc768bbcf35" Nov 03 02:39:05.037078 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:05.037058 5463 scope.go:122] "RemoveContainer" containerID="0354686078f514d804a4f824641bf6f01d75020b6bf11b89a176ddc768bbcf35" Nov 03 02:39:05.037344 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:39:05.037319 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0354686078f514d804a4f824641bf6f01d75020b6bf11b89a176ddc768bbcf35\": container with ID starting with 0354686078f514d804a4f824641bf6f01d75020b6bf11b89a176ddc768bbcf35 not found: ID does not exist" containerID="0354686078f514d804a4f824641bf6f01d75020b6bf11b89a176ddc768bbcf35" Nov 03 02:39:05.037405 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:05.037354 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0354686078f514d804a4f824641bf6f01d75020b6bf11b89a176ddc768bbcf35"} err="failed to get container status \"0354686078f514d804a4f824641bf6f01d75020b6bf11b89a176ddc768bbcf35\": rpc error: code = NotFound desc = could not find container \"0354686078f514d804a4f824641bf6f01d75020b6bf11b89a176ddc768bbcf35\": container with ID starting with 0354686078f514d804a4f824641bf6f01d75020b6bf11b89a176ddc768bbcf35 not found: ID does not exist" Nov 03 02:39:05.192277 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:05.192243 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf"] Nov 03 02:39:05.195591 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:05.195568 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-9tljf"] Nov 03 02:39:06.445123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:06.445089 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" path="/var/lib/kubelet/pods/a9db59ca-ab6b-4c1d-b24c-bcea3efc4268/volumes" Nov 03 02:39:08.757804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:08.757768 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:09.171607 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:09.171574 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr"] Nov 03 02:39:09.171829 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:09.171801 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" podUID="87ab8925-cc59-4bf1-a804-051b4f7907b3" containerName="istio-proxy" containerID="cri-o://58c282afb041c49feffa8b9b50400873fc9c6eb1555a86997d4c7357ae50c88c" gracePeriod=30 Nov 03 02:39:09.220852 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:09.220822 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d"] Nov 03 02:39:09.221042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:09.221019 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" podUID="12d94af3-17e1-4b9e-b27e-1810694acc49" containerName="istio-proxy" containerID="cri-o://43c6716a3d00202658b6cabe79614d0f34f8aea848422fabf0bd41559d3a04af" gracePeriod=30 Nov 03 02:39:09.224457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:09.224433 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz"] Nov 03 02:39:09.224624 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:09.224606 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" podUID="ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" containerName="istio-proxy" containerID="cri-o://3f99e93d1f9a326ed9aab31255056864ceaaf63f136e7cebe0a6cd99b2046e0f" gracePeriod=30 Nov 03 02:39:09.279359 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:09.279321 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6"] Nov 03 02:39:09.279581 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:09.279551 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" podUID="e094c575-bfcd-41a1-902a-a348adde1c14" containerName="istio-proxy" containerID="cri-o://8a937aee1f9c978e6a005b16707d455d10c831ecd5a4820b0b8dcfbd8f41337e" gracePeriod=30 Nov 03 02:39:13.110918 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:39:13.110875 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:39:13.247283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:13.247244 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:13.555668 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:13.555595 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:14.548222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.547204 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:39:14.649093 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.649067 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:39:14.652265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.652243 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:39:14.660808 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.660782 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-data\") pod \"87ab8925-cc59-4bf1-a804-051b4f7907b3\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " Nov 03 02:39:14.660975 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.660866 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-workload-certs\") pod \"87ab8925-cc59-4bf1-a804-051b4f7907b3\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " Nov 03 02:39:14.660975 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.660936 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-podinfo\") pod \"87ab8925-cc59-4bf1-a804-051b4f7907b3\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " Nov 03 02:39:14.661153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.660990 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ab8925-cc59-4bf1-a804-051b4f7907b3-kube-api-access-m2m6w\" (UniqueName: \"kubernetes.io/projected/87ab8925-cc59-4bf1-a804-051b4f7907b3-kube-api-access-m2m6w\") pod \"87ab8925-cc59-4bf1-a804-051b4f7907b3\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " Nov 03 02:39:14.661153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.661040 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87ab8925-cc59-4bf1-a804-051b4f7907b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ab8925-cc59-4bf1-a804-051b4f7907b3-istiod-ca-cert\") pod \"87ab8925-cc59-4bf1-a804-051b4f7907b3\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " Nov 03 02:39:14.661153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.661060 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-data" pod "87ab8925-cc59-4bf1-a804-051b4f7907b3" (UID: "87ab8925-cc59-4bf1-a804-051b4f7907b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:14.661153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.661110 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-token\" (UniqueName: \"kubernetes.io/projected/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-token\") pod \"87ab8925-cc59-4bf1-a804-051b4f7907b3\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " Nov 03 02:39:14.661153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.661139 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-envoy\") pod \"87ab8925-cc59-4bf1-a804-051b4f7907b3\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " Nov 03 02:39:14.661805 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.661310 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-credential-socket\") pod \"87ab8925-cc59-4bf1-a804-051b4f7907b3\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " Nov 03 02:39:14.661805 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.661355 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-workload-socket\") pod \"87ab8925-cc59-4bf1-a804-051b4f7907b3\" (UID: \"87ab8925-cc59-4bf1-a804-051b4f7907b3\") " Nov 03 02:39:14.662116 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.662093 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.662409 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.662337 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-workload-socket" pod "87ab8925-cc59-4bf1-a804-051b4f7907b3" (UID: "87ab8925-cc59-4bf1-a804-051b4f7907b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:14.662994 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.662952 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-workload-certs" pod "87ab8925-cc59-4bf1-a804-051b4f7907b3" (UID: "87ab8925-cc59-4bf1-a804-051b4f7907b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:14.663122 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.663018 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ab8925-cc59-4bf1-a804-051b4f7907b3-istiod-ca-cert" pod "87ab8925-cc59-4bf1-a804-051b4f7907b3" (UID: "87ab8925-cc59-4bf1-a804-051b4f7907b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:39:14.663258 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.663235 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-credential-socket" pod "87ab8925-cc59-4bf1-a804-051b4f7907b3" (UID: "87ab8925-cc59-4bf1-a804-051b4f7907b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:14.671280 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.671246 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ab8925-cc59-4bf1-a804-051b4f7907b3-kube-api-access-m2m6w" pod "87ab8925-cc59-4bf1-a804-051b4f7907b3" (UID: "87ab8925-cc59-4bf1-a804-051b4f7907b3"). InnerVolumeSpecName "kube-api-access-m2m6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:14.671576 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.671556 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-token" pod "87ab8925-cc59-4bf1-a804-051b4f7907b3" (UID: "87ab8925-cc59-4bf1-a804-051b4f7907b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:14.671703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.671675 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-podinfo" pod "87ab8925-cc59-4bf1-a804-051b4f7907b3" (UID: "87ab8925-cc59-4bf1-a804-051b4f7907b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:39:14.673683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.673661 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-envoy" pod "87ab8925-cc59-4bf1-a804-051b4f7907b3" (UID: "87ab8925-cc59-4bf1-a804-051b4f7907b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:14.691258 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.691229 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:39:14.763579 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.763516 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-token\" (UniqueName: \"kubernetes.io/projected/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-token\") pod \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " Nov 03 02:39:14.763579 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.763561 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-credential-socket\") pod \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " Nov 03 02:39:14.763800 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.763604 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-kube-api-access-kxmvq\" (UniqueName: \"kubernetes.io/projected/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-kube-api-access-kxmvq\") pod \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " Nov 03 02:39:14.763800 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.763639 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-data\") pod \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " Nov 03 02:39:14.763800 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.763681 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-envoy\") pod \"12d94af3-17e1-4b9e-b27e-1810694acc49\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " Nov 03 02:39:14.763800 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.763763 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-podinfo\") pod \"12d94af3-17e1-4b9e-b27e-1810694acc49\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " Nov 03 02:39:14.764046 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.763823 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-envoy\") pod \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " Nov 03 02:39:14.764046 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.763877 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12d94af3-17e1-4b9e-b27e-1810694acc49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d94af3-17e1-4b9e-b27e-1810694acc49-istiod-ca-cert\") pod \"12d94af3-17e1-4b9e-b27e-1810694acc49\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " Nov 03 02:39:14.764046 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.763905 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e094c575-bfcd-41a1-902a-a348adde1c14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e094c575-bfcd-41a1-902a-a348adde1c14-istiod-ca-cert\") pod \"e094c575-bfcd-41a1-902a-a348adde1c14\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " Nov 03 02:39:14.764046 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.763940 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e094c575-bfcd-41a1-902a-a348adde1c14-kube-api-access-gzmnk\" (UniqueName: \"kubernetes.io/projected/e094c575-bfcd-41a1-902a-a348adde1c14-kube-api-access-gzmnk\") pod \"e094c575-bfcd-41a1-902a-a348adde1c14\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " Nov 03 02:39:14.764046 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.763964 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-credential-socket" pod "ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" (UID: "ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:14.764046 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764004 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-workload-socket\") pod \"12d94af3-17e1-4b9e-b27e-1810694acc49\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " Nov 03 02:39:14.764415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764050 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-istio-envoy\") pod \"e094c575-bfcd-41a1-902a-a348adde1c14\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " Nov 03 02:39:14.764415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764084 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-istio-data\") pod \"e094c575-bfcd-41a1-902a-a348adde1c14\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " Nov 03 02:39:14.764415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764123 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-workload-socket\") pod \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " Nov 03 02:39:14.764415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764154 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-token\" (UniqueName: \"kubernetes.io/projected/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-token\") pod \"12d94af3-17e1-4b9e-b27e-1810694acc49\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " Nov 03 02:39:14.764415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764183 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-workload-certs\") pod \"e094c575-bfcd-41a1-902a-a348adde1c14\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " Nov 03 02:39:14.764415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764245 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-workload-socket\") pod \"e094c575-bfcd-41a1-902a-a348adde1c14\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " Nov 03 02:39:14.764415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764285 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e094c575-bfcd-41a1-902a-a348adde1c14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e094c575-bfcd-41a1-902a-a348adde1c14-istio-podinfo\") pod \"e094c575-bfcd-41a1-902a-a348adde1c14\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " Nov 03 02:39:14.764415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764326 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-workload-certs\") pod \"12d94af3-17e1-4b9e-b27e-1810694acc49\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " Nov 03 02:39:14.764415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764328 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d94af3-17e1-4b9e-b27e-1810694acc49-istiod-ca-cert" pod "12d94af3-17e1-4b9e-b27e-1810694acc49" (UID: "12d94af3-17e1-4b9e-b27e-1810694acc49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:39:14.764415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764359 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d94af3-17e1-4b9e-b27e-1810694acc49-kube-api-access-r5j2m\" (UniqueName: \"kubernetes.io/projected/12d94af3-17e1-4b9e-b27e-1810694acc49-kube-api-access-r5j2m\") pod \"12d94af3-17e1-4b9e-b27e-1810694acc49\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " Nov 03 02:39:14.765047 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764458 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-credential-socket\") pod \"12d94af3-17e1-4b9e-b27e-1810694acc49\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " Nov 03 02:39:14.765047 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764516 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-data" pod "ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" (UID: "ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:14.765047 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764529 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e094c575-bfcd-41a1-902a-a348adde1c14-istio-token\" (UniqueName: \"kubernetes.io/projected/e094c575-bfcd-41a1-902a-a348adde1c14-istio-token\") pod \"e094c575-bfcd-41a1-902a-a348adde1c14\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " Nov 03 02:39:14.765047 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764577 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-data\") pod \"12d94af3-17e1-4b9e-b27e-1810694acc49\" (UID: \"12d94af3-17e1-4b9e-b27e-1810694acc49\") " Nov 03 02:39:14.765047 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764614 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-podinfo\") pod \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " Nov 03 02:39:14.765047 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764701 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-workload-certs\") pod \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " Nov 03 02:39:14.765047 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764741 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-credential-socket\") pod \"e094c575-bfcd-41a1-902a-a348adde1c14\" (UID: \"e094c575-bfcd-41a1-902a-a348adde1c14\") " Nov 03 02:39:14.765047 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764778 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istiod-ca-cert\") pod \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\" (UID: \"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8\") " Nov 03 02:39:14.765663 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.765638 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.765744 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.765673 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.765744 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.765690 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.765744 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.765707 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d94af3-17e1-4b9e-b27e-1810694acc49-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.765744 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.765722 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2m6w\" (UniqueName: \"kubernetes.io/projected/87ab8925-cc59-4bf1-a804-051b4f7907b3-kube-api-access-m2m6w\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.765744 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.765740 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ab8925-cc59-4bf1-a804-051b4f7907b3-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.766017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.765755 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.766017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.765768 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.766017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.765782 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.766017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.765797 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ab8925-cc59-4bf1-a804-051b4f7907b3-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.766017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.765811 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.766017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764580 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-workload-socket" pod "ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" (UID: "ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:14.766017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.764785 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-workload-socket" pod "12d94af3-17e1-4b9e-b27e-1810694acc49" (UID: "12d94af3-17e1-4b9e-b27e-1810694acc49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:14.766017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.765095 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e094c575-bfcd-41a1-902a-a348adde1c14-istiod-ca-cert" pod "e094c575-bfcd-41a1-902a-a348adde1c14" (UID: "e094c575-bfcd-41a1-902a-a348adde1c14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:39:14.766017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.765649 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-istio-data" pod "e094c575-bfcd-41a1-902a-a348adde1c14" (UID: "e094c575-bfcd-41a1-902a-a348adde1c14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:14.766017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.765818 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-credential-socket" pod "12d94af3-17e1-4b9e-b27e-1810694acc49" (UID: "12d94af3-17e1-4b9e-b27e-1810694acc49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:14.769986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.766163 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istiod-ca-cert" pod "ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" (UID: "ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:39:14.769986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.766533 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-workload-socket" pod "e094c575-bfcd-41a1-902a-a348adde1c14" (UID: "e094c575-bfcd-41a1-902a-a348adde1c14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:14.769986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.766771 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-workload-certs" pod "12d94af3-17e1-4b9e-b27e-1810694acc49" (UID: "12d94af3-17e1-4b9e-b27e-1810694acc49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:14.769986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.766873 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-kube-api-access-kxmvq" pod "ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" (UID: "ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8"). InnerVolumeSpecName "kube-api-access-kxmvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:14.769986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.767348 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-data" pod "12d94af3-17e1-4b9e-b27e-1810694acc49" (UID: "12d94af3-17e1-4b9e-b27e-1810694acc49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:14.769986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.767394 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-workload-certs" pod "ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" (UID: "ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:14.769986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.767659 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-workload-certs" pod "e094c575-bfcd-41a1-902a-a348adde1c14" (UID: "e094c575-bfcd-41a1-902a-a348adde1c14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:14.769986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.767749 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-credential-socket" pod "e094c575-bfcd-41a1-902a-a348adde1c14" (UID: "e094c575-bfcd-41a1-902a-a348adde1c14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:14.769986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.767803 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-istio-envoy" pod "e094c575-bfcd-41a1-902a-a348adde1c14" (UID: "e094c575-bfcd-41a1-902a-a348adde1c14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:14.771117 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.770843 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-envoy" pod "12d94af3-17e1-4b9e-b27e-1810694acc49" (UID: "12d94af3-17e1-4b9e-b27e-1810694acc49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:14.771117 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.770913 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e094c575-bfcd-41a1-902a-a348adde1c14-istio-podinfo" pod "e094c575-bfcd-41a1-902a-a348adde1c14" (UID: "e094c575-bfcd-41a1-902a-a348adde1c14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:39:14.771117 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.771032 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-podinfo" pod "12d94af3-17e1-4b9e-b27e-1810694acc49" (UID: "12d94af3-17e1-4b9e-b27e-1810694acc49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:39:14.771117 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.771037 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e094c575-bfcd-41a1-902a-a348adde1c14-istio-token" pod "e094c575-bfcd-41a1-902a-a348adde1c14" (UID: "e094c575-bfcd-41a1-902a-a348adde1c14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:14.771117 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.771082 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-podinfo" pod "ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" (UID: "ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:39:14.771563 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.771179 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-token" pod "ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" (UID: "ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:14.771563 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.771316 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-token" pod "12d94af3-17e1-4b9e-b27e-1810694acc49" (UID: "12d94af3-17e1-4b9e-b27e-1810694acc49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:14.772090 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.772058 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e094c575-bfcd-41a1-902a-a348adde1c14-kube-api-access-gzmnk" pod "e094c575-bfcd-41a1-902a-a348adde1c14" (UID: "e094c575-bfcd-41a1-902a-a348adde1c14"). InnerVolumeSpecName "kube-api-access-gzmnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:14.772489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.772467 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-envoy" pod "ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" (UID: "ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:14.772753 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.772724 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d94af3-17e1-4b9e-b27e-1810694acc49-kube-api-access-r5j2m" pod "12d94af3-17e1-4b9e-b27e-1810694acc49" (UID: "12d94af3-17e1-4b9e-b27e-1810694acc49"). InnerVolumeSpecName "kube-api-access-r5j2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:14.867325 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867291 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzmnk\" (UniqueName: \"kubernetes.io/projected/e094c575-bfcd-41a1-902a-a348adde1c14-kube-api-access-gzmnk\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.867325 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867326 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.867538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867344 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.867538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867358 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.867538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867371 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.867538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867385 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.867538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867398 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.867538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867412 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.867538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867425 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e094c575-bfcd-41a1-902a-a348adde1c14-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.867538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867440 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.867538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867454 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5j2m\" (UniqueName: \"kubernetes.io/projected/12d94af3-17e1-4b9e-b27e-1810694acc49-kube-api-access-r5j2m\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.867538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867467 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.867538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867481 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e094c575-bfcd-41a1-902a-a348adde1c14-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.867538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867494 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.867538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867509 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.867538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867523 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.867538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867536 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e094c575-bfcd-41a1-902a-a348adde1c14-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.868035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867549 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.868035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867564 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.868035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867577 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxmvq\" (UniqueName: \"kubernetes.io/projected/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-kube-api-access-kxmvq\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.868035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867590 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.868035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867604 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d94af3-17e1-4b9e-b27e-1810694acc49-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.868035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867617 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:14.868035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:14.867631 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e094c575-bfcd-41a1-902a-a348adde1c14-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:15.148049 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.148020 5463 generic.go:356] "Generic (PLEG): container finished" podID="e094c575-bfcd-41a1-902a-a348adde1c14" containerID="8a937aee1f9c978e6a005b16707d455d10c831ecd5a4820b0b8dcfbd8f41337e" exitCode=0 Nov 03 02:39:15.148244 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.148069 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" Nov 03 02:39:15.148244 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.148096 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" event={"ID":"e094c575-bfcd-41a1-902a-a348adde1c14","Type":"ContainerDied","Data":"8a937aee1f9c978e6a005b16707d455d10c831ecd5a4820b0b8dcfbd8f41337e"} Nov 03 02:39:15.148244 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.148154 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6" event={"ID":"e094c575-bfcd-41a1-902a-a348adde1c14","Type":"ContainerDied","Data":"03a55ea5b09ceb94d5b04feeeddc0addc715b706f9d3dd6177e91106acf5a641"} Nov 03 02:39:15.148244 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.148177 5463 scope.go:122] "RemoveContainer" containerID="8a937aee1f9c978e6a005b16707d455d10c831ecd5a4820b0b8dcfbd8f41337e" Nov 03 02:39:15.149853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.149828 5463 generic.go:356] "Generic (PLEG): container finished" podID="ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" containerID="3f99e93d1f9a326ed9aab31255056864ceaaf63f136e7cebe0a6cd99b2046e0f" exitCode=0 Nov 03 02:39:15.149999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.149869 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" Nov 03 02:39:15.149999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.149923 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" event={"ID":"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8","Type":"ContainerDied","Data":"3f99e93d1f9a326ed9aab31255056864ceaaf63f136e7cebe0a6cd99b2046e0f"} Nov 03 02:39:15.149999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.149987 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz" event={"ID":"ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8","Type":"ContainerDied","Data":"ded5eeebde6f682b676cce49ae4562ec25dbb78b923d2c5e4236297682aabe39"} Nov 03 02:39:15.151657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.151629 5463 generic.go:356] "Generic (PLEG): container finished" podID="87ab8925-cc59-4bf1-a804-051b4f7907b3" containerID="58c282afb041c49feffa8b9b50400873fc9c6eb1555a86997d4c7357ae50c88c" exitCode=0 Nov 03 02:39:15.151774 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.151679 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" Nov 03 02:39:15.151774 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.151711 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" event={"ID":"87ab8925-cc59-4bf1-a804-051b4f7907b3","Type":"ContainerDied","Data":"58c282afb041c49feffa8b9b50400873fc9c6eb1555a86997d4c7357ae50c88c"} Nov 03 02:39:15.151774 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.151764 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr" event={"ID":"87ab8925-cc59-4bf1-a804-051b4f7907b3","Type":"ContainerDied","Data":"06a118f4a78e2dc0415fa45ea7c2721cf441877f2822cb8c11cc873407d131e6"} Nov 03 02:39:15.153386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.153364 5463 generic.go:356] "Generic (PLEG): container finished" podID="12d94af3-17e1-4b9e-b27e-1810694acc49" containerID="43c6716a3d00202658b6cabe79614d0f34f8aea848422fabf0bd41559d3a04af" exitCode=0 Nov 03 02:39:15.153487 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.153401 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" event={"ID":"12d94af3-17e1-4b9e-b27e-1810694acc49","Type":"ContainerDied","Data":"43c6716a3d00202658b6cabe79614d0f34f8aea848422fabf0bd41559d3a04af"} Nov 03 02:39:15.153487 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.153415 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" Nov 03 02:39:15.153487 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.153437 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d" event={"ID":"12d94af3-17e1-4b9e-b27e-1810694acc49","Type":"ContainerDied","Data":"f94d4e33b8f1cc14437d3cdb429cff610477e0f0b1fb9486aa9f361a2218ed5e"} Nov 03 02:39:15.159855 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.159835 5463 scope.go:122] "RemoveContainer" containerID="628f71f8b1f9ef99eb0408f733671b4fad509b7d267d2d5d1672b3ae54f2c8c8" Nov 03 02:39:15.171585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.171568 5463 scope.go:122] "RemoveContainer" containerID="8a937aee1f9c978e6a005b16707d455d10c831ecd5a4820b0b8dcfbd8f41337e" Nov 03 02:39:15.171863 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:39:15.171845 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a937aee1f9c978e6a005b16707d455d10c831ecd5a4820b0b8dcfbd8f41337e\": container with ID starting with 8a937aee1f9c978e6a005b16707d455d10c831ecd5a4820b0b8dcfbd8f41337e not found: ID does not exist" containerID="8a937aee1f9c978e6a005b16707d455d10c831ecd5a4820b0b8dcfbd8f41337e" Nov 03 02:39:15.171913 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.171869 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a937aee1f9c978e6a005b16707d455d10c831ecd5a4820b0b8dcfbd8f41337e"} err="failed to get container status \"8a937aee1f9c978e6a005b16707d455d10c831ecd5a4820b0b8dcfbd8f41337e\": rpc error: code = NotFound desc = could not find container \"8a937aee1f9c978e6a005b16707d455d10c831ecd5a4820b0b8dcfbd8f41337e\": container with ID starting with 8a937aee1f9c978e6a005b16707d455d10c831ecd5a4820b0b8dcfbd8f41337e not found: ID does not exist" Nov 03 02:39:15.171913 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.171884 5463 scope.go:122] "RemoveContainer" containerID="628f71f8b1f9ef99eb0408f733671b4fad509b7d267d2d5d1672b3ae54f2c8c8" Nov 03 02:39:15.172132 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:39:15.172118 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"628f71f8b1f9ef99eb0408f733671b4fad509b7d267d2d5d1672b3ae54f2c8c8\": container with ID starting with 628f71f8b1f9ef99eb0408f733671b4fad509b7d267d2d5d1672b3ae54f2c8c8 not found: ID does not exist" containerID="628f71f8b1f9ef99eb0408f733671b4fad509b7d267d2d5d1672b3ae54f2c8c8" Nov 03 02:39:15.172176 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.172136 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"628f71f8b1f9ef99eb0408f733671b4fad509b7d267d2d5d1672b3ae54f2c8c8"} err="failed to get container status \"628f71f8b1f9ef99eb0408f733671b4fad509b7d267d2d5d1672b3ae54f2c8c8\": rpc error: code = NotFound desc = could not find container \"628f71f8b1f9ef99eb0408f733671b4fad509b7d267d2d5d1672b3ae54f2c8c8\": container with ID starting with 628f71f8b1f9ef99eb0408f733671b4fad509b7d267d2d5d1672b3ae54f2c8c8 not found: ID does not exist" Nov 03 02:39:15.172176 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.172148 5463 scope.go:122] "RemoveContainer" containerID="3f99e93d1f9a326ed9aab31255056864ceaaf63f136e7cebe0a6cd99b2046e0f" Nov 03 02:39:15.181102 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.181086 5463 scope.go:122] "RemoveContainer" containerID="3f99e93d1f9a326ed9aab31255056864ceaaf63f136e7cebe0a6cd99b2046e0f" Nov 03 02:39:15.181433 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:39:15.181413 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f99e93d1f9a326ed9aab31255056864ceaaf63f136e7cebe0a6cd99b2046e0f\": container with ID starting with 3f99e93d1f9a326ed9aab31255056864ceaaf63f136e7cebe0a6cd99b2046e0f not found: ID does not exist" containerID="3f99e93d1f9a326ed9aab31255056864ceaaf63f136e7cebe0a6cd99b2046e0f" Nov 03 02:39:15.181510 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.181437 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f99e93d1f9a326ed9aab31255056864ceaaf63f136e7cebe0a6cd99b2046e0f"} err="failed to get container status \"3f99e93d1f9a326ed9aab31255056864ceaaf63f136e7cebe0a6cd99b2046e0f\": rpc error: code = NotFound desc = could not find container \"3f99e93d1f9a326ed9aab31255056864ceaaf63f136e7cebe0a6cd99b2046e0f\": container with ID starting with 3f99e93d1f9a326ed9aab31255056864ceaaf63f136e7cebe0a6cd99b2046e0f not found: ID does not exist" Nov 03 02:39:15.181510 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.181452 5463 scope.go:122] "RemoveContainer" containerID="58c282afb041c49feffa8b9b50400873fc9c6eb1555a86997d4c7357ae50c88c" Nov 03 02:39:15.190413 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.190398 5463 scope.go:122] "RemoveContainer" containerID="58c282afb041c49feffa8b9b50400873fc9c6eb1555a86997d4c7357ae50c88c" Nov 03 02:39:15.190651 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:39:15.190631 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58c282afb041c49feffa8b9b50400873fc9c6eb1555a86997d4c7357ae50c88c\": container with ID starting with 58c282afb041c49feffa8b9b50400873fc9c6eb1555a86997d4c7357ae50c88c not found: ID does not exist" containerID="58c282afb041c49feffa8b9b50400873fc9c6eb1555a86997d4c7357ae50c88c" Nov 03 02:39:15.190725 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.190660 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58c282afb041c49feffa8b9b50400873fc9c6eb1555a86997d4c7357ae50c88c"} err="failed to get container status \"58c282afb041c49feffa8b9b50400873fc9c6eb1555a86997d4c7357ae50c88c\": rpc error: code = NotFound desc = could not find container \"58c282afb041c49feffa8b9b50400873fc9c6eb1555a86997d4c7357ae50c88c\": container with ID starting with 58c282afb041c49feffa8b9b50400873fc9c6eb1555a86997d4c7357ae50c88c not found: ID does not exist" Nov 03 02:39:15.190725 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.190676 5463 scope.go:122] "RemoveContainer" containerID="43c6716a3d00202658b6cabe79614d0f34f8aea848422fabf0bd41559d3a04af" Nov 03 02:39:15.230138 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.230117 5463 scope.go:122] "RemoveContainer" containerID="43c6716a3d00202658b6cabe79614d0f34f8aea848422fabf0bd41559d3a04af" Nov 03 02:39:15.230449 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:39:15.230428 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43c6716a3d00202658b6cabe79614d0f34f8aea848422fabf0bd41559d3a04af\": container with ID starting with 43c6716a3d00202658b6cabe79614d0f34f8aea848422fabf0bd41559d3a04af not found: ID does not exist" containerID="43c6716a3d00202658b6cabe79614d0f34f8aea848422fabf0bd41559d3a04af" Nov 03 02:39:15.230504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.230456 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43c6716a3d00202658b6cabe79614d0f34f8aea848422fabf0bd41559d3a04af"} err="failed to get container status \"43c6716a3d00202658b6cabe79614d0f34f8aea848422fabf0bd41559d3a04af\": rpc error: code = NotFound desc = could not find container \"43c6716a3d00202658b6cabe79614d0f34f8aea848422fabf0bd41559d3a04af\": container with ID starting with 43c6716a3d00202658b6cabe79614d0f34f8aea848422fabf0bd41559d3a04af not found: ID does not exist" Nov 03 02:39:15.315849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.315817 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6"] Nov 03 02:39:15.319236 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.319207 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-9kgr6"] Nov 03 02:39:15.428636 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.428563 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d"] Nov 03 02:39:15.432545 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.432521 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-5vf5d"] Nov 03 02:39:15.543386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.543360 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz"] Nov 03 02:39:15.546796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.546771 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-s2jqz"] Nov 03 02:39:15.656454 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.656421 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr"] Nov 03 02:39:15.660897 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:15.660863 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-cnltr"] Nov 03 02:39:16.443757 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:16.443720 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12d94af3-17e1-4b9e-b27e-1810694acc49" path="/var/lib/kubelet/pods/12d94af3-17e1-4b9e-b27e-1810694acc49/volumes" Nov 03 02:39:16.444252 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:16.444237 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87ab8925-cc59-4bf1-a804-051b4f7907b3" path="/var/lib/kubelet/pods/87ab8925-cc59-4bf1-a804-051b4f7907b3/volumes" Nov 03 02:39:16.444669 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:16.444657 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" path="/var/lib/kubelet/pods/ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8/volumes" Nov 03 02:39:16.445402 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:16.445386 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e094c575-bfcd-41a1-902a-a348adde1c14" path="/var/lib/kubelet/pods/e094c575-bfcd-41a1-902a-a348adde1c14/volumes" Nov 03 02:39:18.310062 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:18.310031 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:18.871045 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:18.871007 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:22.982379 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:22.982339 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:27.647105 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:27.647066 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:32.437206 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:32.437149 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:37.221647 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.221606 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:37.711377 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.711343 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd"] Nov 03 02:39:37.712019 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712004 5463 cpu_manager.go:414] "containerMap: removing container" podUID="4d0ba391-497c-4575-909a-10a65432678d" containerName="istio-proxy" Nov 03 02:39:37.712072 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712021 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d0ba391-497c-4575-909a-10a65432678d" containerName="istio-proxy" Nov 03 02:39:37.712072 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712036 5463 cpu_manager.go:414] "containerMap: removing container" podUID="e094c575-bfcd-41a1-902a-a348adde1c14" containerName="istio-proxy" Nov 03 02:39:37.712072 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712041 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e094c575-bfcd-41a1-902a-a348adde1c14" containerName="istio-proxy" Nov 03 02:39:37.712072 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712053 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" containerName="istio-proxy" Nov 03 02:39:37.712072 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712058 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" containerName="istio-proxy" Nov 03 02:39:37.712072 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712068 5463 cpu_manager.go:414] "containerMap: removing container" podUID="12d94af3-17e1-4b9e-b27e-1810694acc49" containerName="istio-proxy" Nov 03 02:39:37.712072 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712073 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d94af3-17e1-4b9e-b27e-1810694acc49" containerName="istio-proxy" Nov 03 02:39:37.712313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712085 5463 cpu_manager.go:414] "containerMap: removing container" podUID="ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" containerName="istio-proxy" Nov 03 02:39:37.712313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712090 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" containerName="istio-proxy" Nov 03 02:39:37.712313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712096 5463 cpu_manager.go:414] "containerMap: removing container" podUID="e094c575-bfcd-41a1-902a-a348adde1c14" containerName="istio-proxy" Nov 03 02:39:37.712313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712102 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e094c575-bfcd-41a1-902a-a348adde1c14" containerName="istio-proxy" Nov 03 02:39:37.712313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712110 5463 cpu_manager.go:414] "containerMap: removing container" podUID="87ab8925-cc59-4bf1-a804-051b4f7907b3" containerName="istio-proxy" Nov 03 02:39:37.712313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712115 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ab8925-cc59-4bf1-a804-051b4f7907b3" containerName="istio-proxy" Nov 03 02:39:37.712313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712240 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d0ba391-497c-4575-909a-10a65432678d" containerName="istio-proxy" Nov 03 02:39:37.712313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712248 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e094c575-bfcd-41a1-902a-a348adde1c14" containerName="istio-proxy" Nov 03 02:39:37.712313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712257 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9db59ca-ab6b-4c1d-b24c-bcea3efc4268" containerName="istio-proxy" Nov 03 02:39:37.712313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712263 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d94af3-17e1-4b9e-b27e-1810694acc49" containerName="istio-proxy" Nov 03 02:39:37.712313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712271 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce481b0b-ac96-4898-bcfd-e1afa0a3bcb8" containerName="istio-proxy" Nov 03 02:39:37.712313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712277 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e094c575-bfcd-41a1-902a-a348adde1c14" containerName="istio-proxy" Nov 03 02:39:37.712313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.712281 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ab8925-cc59-4bf1-a804-051b4f7907b3" containerName="istio-proxy" Nov 03 02:39:37.715805 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.715787 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:37.769025 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.768988 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd"] Nov 03 02:39:37.790545 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.790509 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c87a029-189f-4e51-91dd-4e69ea57f45b-tls-certs\") pod \"praxis-route-f954db-kserve-6bffb75fd5-lxgbd\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:37.790746 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.790654 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-model-cache\") pod \"praxis-route-f954db-kserve-6bffb75fd5-lxgbd\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:37.790746 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.790679 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-home\") pod \"praxis-route-f954db-kserve-6bffb75fd5-lxgbd\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:37.790878 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.790744 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-tmp-dir\") pod \"praxis-route-f954db-kserve-6bffb75fd5-lxgbd\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:37.790878 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.790820 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kb85\" (UniqueName: \"kubernetes.io/projected/8c87a029-189f-4e51-91dd-4e69ea57f45b-kube-api-access-6kb85\") pod \"praxis-route-f954db-kserve-6bffb75fd5-lxgbd\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:37.790878 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.790839 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-dshm\") pod \"praxis-route-f954db-kserve-6bffb75fd5-lxgbd\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:37.815253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.815216 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-f954db\"/\"openshift-service-ca.crt\"" Nov 03 02:39:37.815253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.815227 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-f954db\"/\"kube-root-ca.crt\"" Nov 03 02:39:37.818379 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.818342 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-f954db\"/\"praxis-route-f954db-kserve-self-signed-certs\"" Nov 03 02:39:37.818526 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.818482 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-f954db\"/\"default-dockercfg-q7ppl\"" Nov 03 02:39:37.825319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.825297 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:37.892460 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.892431 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-model-cache\") pod \"praxis-route-f954db-kserve-6bffb75fd5-lxgbd\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:37.892627 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.892466 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-home\") pod \"praxis-route-f954db-kserve-6bffb75fd5-lxgbd\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:37.892627 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.892540 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-tmp-dir\") pod \"praxis-route-f954db-kserve-6bffb75fd5-lxgbd\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:37.892737 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.892626 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kb85\" (UniqueName: \"kubernetes.io/projected/8c87a029-189f-4e51-91dd-4e69ea57f45b-kube-api-access-6kb85\") pod \"praxis-route-f954db-kserve-6bffb75fd5-lxgbd\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:37.892737 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.892678 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-dshm\") pod \"praxis-route-f954db-kserve-6bffb75fd5-lxgbd\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:37.892892 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.892873 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c87a029-189f-4e51-91dd-4e69ea57f45b-tls-certs\") pod \"praxis-route-f954db-kserve-6bffb75fd5-lxgbd\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:37.892993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.892907 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-model-cache\") pod \"praxis-route-f954db-kserve-6bffb75fd5-lxgbd\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:37.892993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.892948 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-home\") pod \"praxis-route-f954db-kserve-6bffb75fd5-lxgbd\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:37.893183 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.893157 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-tmp-dir\") pod \"praxis-route-f954db-kserve-6bffb75fd5-lxgbd\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:37.895379 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.895357 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-dshm\") pod \"praxis-route-f954db-kserve-6bffb75fd5-lxgbd\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:37.895483 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.895431 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c87a029-189f-4e51-91dd-4e69ea57f45b-tls-certs\") pod \"praxis-route-f954db-kserve-6bffb75fd5-lxgbd\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:37.949926 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:37.949900 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kb85\" (UniqueName: \"kubernetes.io/projected/8c87a029-189f-4e51-91dd-4e69ea57f45b-kube-api-access-6kb85\") pod \"praxis-route-f954db-kserve-6bffb75fd5-lxgbd\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:38.029016 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:38.028938 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:38.404079 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:39:38.404027 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c87a029_189f_4e51_91dd_4e69ea57f45b.slice/crio-e843833d594d006db4a732861a48380adb5fec3b9ad89451275560e32e92c7d7 WatchSource:0}: Error finding container e843833d594d006db4a732861a48380adb5fec3b9ad89451275560e32e92c7d7: Status 404 returned error can't find the container with id e843833d594d006db4a732861a48380adb5fec3b9ad89451275560e32e92c7d7 Nov 03 02:39:38.407827 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:38.407804 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd"] Nov 03 02:39:39.361535 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:39.361483 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" event={"ID":"8c87a029-189f-4e51-91dd-4e69ea57f45b","Type":"ContainerStarted","Data":"518c2f48b4424b85d94c96a73ad5c6c1dc36d09ca7a33c1304d948a9a1b19076"} Nov 03 02:39:39.361735 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:39.361551 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" event={"ID":"8c87a029-189f-4e51-91dd-4e69ea57f45b","Type":"ContainerStarted","Data":"e843833d594d006db4a732861a48380adb5fec3b9ad89451275560e32e92c7d7"} Nov 03 02:39:39.477700 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:39.476415 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" podStartSLOduration=2.476396711 podStartE2EDuration="2.476396711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:39:39.474947195 +0000 UTC m=+2861.159056350" watchObservedRunningTime="2026-11-03 02:39:39.476396711 +0000 UTC m=+2861.160505874" Nov 03 02:39:40.030056 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:40.030016 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:41.950842 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:41.950769 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:43.008528 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:39:43.008494 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:39:46.224002 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:46.223963 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:46.648323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:46.648273 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:48.029604 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:48.029568 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:48.048823 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:48.048794 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:48.444710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:48.444684 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:39:51.456654 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:51.456618 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:51.603787 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:51.603754 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:56.329053 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:39:56.328999 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:00.458252 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:40:00.458222 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:01.060570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:40:01.060524 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:05.076807 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:40:05.076776 5463 scope.go:122] "RemoveContainer" containerID="ef3e2d14c27ab232bf2fe68c4cca7898221fc18170933346aeed63184bf9526a" Nov 03 02:40:05.922665 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:40:05.922594 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:10.704236 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:40:10.704119 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:13.008237 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:40:13.008185 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:40:15.402707 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:40:15.402667 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:19.189601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:40:19.189565 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:20.102423 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:40:20.102382 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:24.819561 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:40:24.819519 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:29.707667 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:40:29.707637 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:34.440549 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:40:34.440514 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:39.226560 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:40:39.226526 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:43.008803 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:40:43.008758 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:40:43.916295 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:40:43.916263 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:48.591562 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:40:48.591526 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:48.794005 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:40:48.793948 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:53.412937 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:40:53.412877 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:58.235464 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:40:58.235431 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:02.855969 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:02.855926 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:07.656574 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:07.656542 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:09.019636 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:09.019601 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:12.437054 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:12.437017 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:13.009811 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:41:13.009774 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:41:15.412732 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:15.412698 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj"] Nov 03 02:41:15.413271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:15.412969 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" podUID="ba4b9eb7-fc25-451c-9e9f-aae89573754d" containerName="istio-proxy" containerID="cri-o://3f52935ca2228ef3aace5345d70c6d77d4931b6d615bbfd33f74049c68ce1682" gracePeriod=30 Nov 03 02:41:17.239805 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:17.239770 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:19.512542 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:19.512510 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:20.443492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.443462 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:20.602080 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.602016 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:20.717312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.717287 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:41:20.778466 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.778429 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-data\") pod \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " Nov 03 02:41:20.778637 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.778524 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba4b9eb7-fc25-451c-9e9f-aae89573754d-kube-api-access-nkgkd\" (UniqueName: \"kubernetes.io/projected/ba4b9eb7-fc25-451c-9e9f-aae89573754d-kube-api-access-nkgkd\") pod \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " Nov 03 02:41:20.778637 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.778593 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-envoy\") pod \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " Nov 03 02:41:20.778741 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.778638 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-podinfo\") pod \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " Nov 03 02:41:20.778741 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.778697 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-workload-socket\") pod \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " Nov 03 02:41:20.778809 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.778770 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-data" pod "ba4b9eb7-fc25-451c-9e9f-aae89573754d" (UID: "ba4b9eb7-fc25-451c-9e9f-aae89573754d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:20.778809 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.778786 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-credential-socket\") pod \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " Nov 03 02:41:20.778896 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.778818 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istiod-ca-cert\") pod \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " Nov 03 02:41:20.778956 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.778941 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-workload-certs\") pod \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " Nov 03 02:41:20.779059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.779042 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-token\" (UniqueName: \"kubernetes.io/projected/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-token\") pod \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\" (UID: \"ba4b9eb7-fc25-451c-9e9f-aae89573754d\") " Nov 03 02:41:20.780015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.779270 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-workload-socket" pod "ba4b9eb7-fc25-451c-9e9f-aae89573754d" (UID: "ba4b9eb7-fc25-451c-9e9f-aae89573754d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:20.780015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.779728 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-workload-certs" pod "ba4b9eb7-fc25-451c-9e9f-aae89573754d" (UID: "ba4b9eb7-fc25-451c-9e9f-aae89573754d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:20.780015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.779841 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-credential-socket" pod "ba4b9eb7-fc25-451c-9e9f-aae89573754d" (UID: "ba4b9eb7-fc25-451c-9e9f-aae89573754d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:20.780015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.779971 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:20.780015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.779991 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:20.780428 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.780028 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istiod-ca-cert" pod "ba4b9eb7-fc25-451c-9e9f-aae89573754d" (UID: "ba4b9eb7-fc25-451c-9e9f-aae89573754d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:41:20.781882 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.781849 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-envoy" pod "ba4b9eb7-fc25-451c-9e9f-aae89573754d" (UID: "ba4b9eb7-fc25-451c-9e9f-aae89573754d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:20.783426 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.783390 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-token" pod "ba4b9eb7-fc25-451c-9e9f-aae89573754d" (UID: "ba4b9eb7-fc25-451c-9e9f-aae89573754d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:20.783546 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.783447 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-podinfo" pod "ba4b9eb7-fc25-451c-9e9f-aae89573754d" (UID: "ba4b9eb7-fc25-451c-9e9f-aae89573754d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:41:20.783719 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.783682 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4b9eb7-fc25-451c-9e9f-aae89573754d-kube-api-access-nkgkd" pod "ba4b9eb7-fc25-451c-9e9f-aae89573754d" (UID: "ba4b9eb7-fc25-451c-9e9f-aae89573754d"). InnerVolumeSpecName "kube-api-access-nkgkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:20.881743 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.881710 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:20.881743 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.881739 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:20.881743 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.881749 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:20.882065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.881759 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:20.882065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.881767 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4b9eb7-fc25-451c-9e9f-aae89573754d-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:20.882065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.881777 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4b9eb7-fc25-451c-9e9f-aae89573754d-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:20.882065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:20.881789 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkgkd\" (UniqueName: \"kubernetes.io/projected/ba4b9eb7-fc25-451c-9e9f-aae89573754d-kube-api-access-nkgkd\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:21.232461 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:21.232422 5463 generic.go:356] "Generic (PLEG): container finished" podID="ba4b9eb7-fc25-451c-9e9f-aae89573754d" containerID="3f52935ca2228ef3aace5345d70c6d77d4931b6d615bbfd33f74049c68ce1682" exitCode=0 Nov 03 02:41:21.232637 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:21.232469 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" Nov 03 02:41:21.232637 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:21.232510 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" event={"ID":"ba4b9eb7-fc25-451c-9e9f-aae89573754d","Type":"ContainerDied","Data":"3f52935ca2228ef3aace5345d70c6d77d4931b6d615bbfd33f74049c68ce1682"} Nov 03 02:41:21.232637 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:21.232569 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj" event={"ID":"ba4b9eb7-fc25-451c-9e9f-aae89573754d","Type":"ContainerDied","Data":"de3cc175f341c60e1682f5c93945d2bbc9ff00c34ac5094ca92bcee6b4016d29"} Nov 03 02:41:21.232637 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:21.232591 5463 scope.go:122] "RemoveContainer" containerID="3f52935ca2228ef3aace5345d70c6d77d4931b6d615bbfd33f74049c68ce1682" Nov 03 02:41:21.244138 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:21.244117 5463 scope.go:122] "RemoveContainer" containerID="3f52935ca2228ef3aace5345d70c6d77d4931b6d615bbfd33f74049c68ce1682" Nov 03 02:41:21.244416 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:41:21.244396 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f52935ca2228ef3aace5345d70c6d77d4931b6d615bbfd33f74049c68ce1682\": container with ID starting with 3f52935ca2228ef3aace5345d70c6d77d4931b6d615bbfd33f74049c68ce1682 not found: ID does not exist" containerID="3f52935ca2228ef3aace5345d70c6d77d4931b6d615bbfd33f74049c68ce1682" Nov 03 02:41:21.244467 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:21.244423 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f52935ca2228ef3aace5345d70c6d77d4931b6d615bbfd33f74049c68ce1682"} err="failed to get container status \"3f52935ca2228ef3aace5345d70c6d77d4931b6d615bbfd33f74049c68ce1682\": rpc error: code = NotFound desc = could not find container \"3f52935ca2228ef3aace5345d70c6d77d4931b6d615bbfd33f74049c68ce1682\": container with ID starting with 3f52935ca2228ef3aace5345d70c6d77d4931b6d615bbfd33f74049c68ce1682 not found: ID does not exist" Nov 03 02:41:21.400686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:21.400650 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj"] Nov 03 02:41:21.451386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:21.451353 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-pbvkj"] Nov 03 02:41:21.981462 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:21.981422 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:22.445215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:22.445167 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba4b9eb7-fc25-451c-9e9f-aae89573754d" path="/var/lib/kubelet/pods/ba4b9eb7-fc25-451c-9e9f-aae89573754d/volumes" Nov 03 02:41:26.647703 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:26.647667 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:31.298271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:31.298147 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:36.021841 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:36.021806 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:40.713632 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:40.713542 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:43.009282 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:41:43.009247 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:41:45.452164 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:45.452124 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:50.220235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:50.220186 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:54.935005 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:54.934968 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:59.707167 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:41:59.707122 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:02.967704 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:02.967672 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:42:02.968979 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:02.968951 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:42:04.420966 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:04.420921 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:09.287183 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:09.287137 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:10.159001 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:10.158968 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:13.009854 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:42:13.009822 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:42:13.922586 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:13.922540 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:18.674037 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:18.673999 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:19.921990 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:19.921949 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:23.367104 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:23.367060 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:28.114114 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:28.114075 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:32.832108 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:32.832052 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:33.333282 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:33.333250 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:37.534705 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:37.534671 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:38.169325 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:38.169291 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd"] Nov 03 02:42:38.259993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:38.259966 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:42.244779 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:42.244746 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:43.007904 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:42:43.007873 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:42:46.947875 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:46.947843 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:51.636664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:51.636614 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:53.201835 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.201745 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" podUID="8c87a029-189f-4e51-91dd-4e69ea57f45b" containerName="main" containerID="cri-o://518c2f48b4424b85d94c96a73ad5c6c1dc36d09ca7a33c1304d948a9a1b19076" gracePeriod=45 Nov 03 02:42:53.589860 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.589826 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:42:53.660457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.660412 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c87a029-189f-4e51-91dd-4e69ea57f45b-tls-certs\" (UniqueName: \"kubernetes.io/secret/8c87a029-189f-4e51-91dd-4e69ea57f45b-tls-certs\") pod \"8c87a029-189f-4e51-91dd-4e69ea57f45b\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " Nov 03 02:42:53.660657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.660582 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-model-cache\") pod \"8c87a029-189f-4e51-91dd-4e69ea57f45b\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " Nov 03 02:42:53.660657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.660653 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-home\" (UniqueName: \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-home\") pod \"8c87a029-189f-4e51-91dd-4e69ea57f45b\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " Nov 03 02:42:53.660783 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.660717 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-tmp-dir\") pod \"8c87a029-189f-4e51-91dd-4e69ea57f45b\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " Nov 03 02:42:53.660865 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.660817 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c87a029-189f-4e51-91dd-4e69ea57f45b-kube-api-access-6kb85\" (UniqueName: \"kubernetes.io/projected/8c87a029-189f-4e51-91dd-4e69ea57f45b-kube-api-access-6kb85\") pod \"8c87a029-189f-4e51-91dd-4e69ea57f45b\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " Nov 03 02:42:53.660865 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.660835 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-model-cache" pod "8c87a029-189f-4e51-91dd-4e69ea57f45b" (UID: "8c87a029-189f-4e51-91dd-4e69ea57f45b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:42:53.660984 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.660915 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-dshm\") pod \"8c87a029-189f-4e51-91dd-4e69ea57f45b\" (UID: \"8c87a029-189f-4e51-91dd-4e69ea57f45b\") " Nov 03 02:42:53.661074 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.660911 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-home" pod "8c87a029-189f-4e51-91dd-4e69ea57f45b" (UID: "8c87a029-189f-4e51-91dd-4e69ea57f45b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:42:53.661353 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.661332 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-tmp-dir" pod "8c87a029-189f-4e51-91dd-4e69ea57f45b" (UID: "8c87a029-189f-4e51-91dd-4e69ea57f45b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:42:53.661545 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.661521 5463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-model-cache\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:42:53.661663 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.661548 5463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-home\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:42:53.661663 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.661564 5463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-tmp-dir\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:42:53.663143 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.663124 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-dshm" pod "8c87a029-189f-4e51-91dd-4e69ea57f45b" (UID: "8c87a029-189f-4e51-91dd-4e69ea57f45b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:42:53.663302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.663284 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c87a029-189f-4e51-91dd-4e69ea57f45b-kube-api-access-6kb85" pod "8c87a029-189f-4e51-91dd-4e69ea57f45b" (UID: "8c87a029-189f-4e51-91dd-4e69ea57f45b"). InnerVolumeSpecName "kube-api-access-6kb85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:42:53.663302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.663290 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c87a029-189f-4e51-91dd-4e69ea57f45b-tls-certs" pod "8c87a029-189f-4e51-91dd-4e69ea57f45b" (UID: "8c87a029-189f-4e51-91dd-4e69ea57f45b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:42:53.763012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.762931 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kb85\" (UniqueName: \"kubernetes.io/projected/8c87a029-189f-4e51-91dd-4e69ea57f45b-kube-api-access-6kb85\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:42:53.763012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.762968 5463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c87a029-189f-4e51-91dd-4e69ea57f45b-dshm\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:42:53.763012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.762983 5463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c87a029-189f-4e51-91dd-4e69ea57f45b-tls-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:42:53.999566 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.999527 5463 generic.go:356] "Generic (PLEG): container finished" podID="8c87a029-189f-4e51-91dd-4e69ea57f45b" containerID="518c2f48b4424b85d94c96a73ad5c6c1dc36d09ca7a33c1304d948a9a1b19076" exitCode=0 Nov 03 02:42:53.999730 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.999571 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" Nov 03 02:42:53.999730 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.999608 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" event={"ID":"8c87a029-189f-4e51-91dd-4e69ea57f45b","Type":"ContainerDied","Data":"518c2f48b4424b85d94c96a73ad5c6c1dc36d09ca7a33c1304d948a9a1b19076"} Nov 03 02:42:53.999730 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.999648 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" event={"ID":"8c87a029-189f-4e51-91dd-4e69ea57f45b","Type":"ContainerDied","Data":"e843833d594d006db4a732861a48380adb5fec3b9ad89451275560e32e92c7d7"} Nov 03 02:42:53.999730 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:53.999669 5463 scope.go:122] "RemoveContainer" containerID="518c2f48b4424b85d94c96a73ad5c6c1dc36d09ca7a33c1304d948a9a1b19076" Nov 03 02:42:54.017523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:54.017503 5463 scope.go:122] "RemoveContainer" containerID="518c2f48b4424b85d94c96a73ad5c6c1dc36d09ca7a33c1304d948a9a1b19076" Nov 03 02:42:54.017831 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:42:54.017802 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"518c2f48b4424b85d94c96a73ad5c6c1dc36d09ca7a33c1304d948a9a1b19076\": container with ID starting with 518c2f48b4424b85d94c96a73ad5c6c1dc36d09ca7a33c1304d948a9a1b19076 not found: ID does not exist" containerID="518c2f48b4424b85d94c96a73ad5c6c1dc36d09ca7a33c1304d948a9a1b19076" Nov 03 02:42:54.017955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:54.017840 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518c2f48b4424b85d94c96a73ad5c6c1dc36d09ca7a33c1304d948a9a1b19076"} err="failed to get container status \"518c2f48b4424b85d94c96a73ad5c6c1dc36d09ca7a33c1304d948a9a1b19076\": rpc error: code = NotFound desc = could not find container \"518c2f48b4424b85d94c96a73ad5c6c1dc36d09ca7a33c1304d948a9a1b19076\": container with ID starting with 518c2f48b4424b85d94c96a73ad5c6c1dc36d09ca7a33c1304d948a9a1b19076 not found: ID does not exist" Nov 03 02:42:54.165041 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:54.165012 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd"] Nov 03 02:42:54.214932 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:54.214898 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd"] Nov 03 02:42:54.431865 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:54.431823 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-f954db/praxis-route-f954db-kserve-6bffb75fd5-lxgbd" podUID="8c87a029-189f-4e51-91dd-4e69ea57f45b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.253:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:42:54.444492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:54.444467 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c87a029-189f-4e51-91dd-4e69ea57f45b" path="/var/lib/kubelet/pods/8c87a029-189f-4e51-91dd-4e69ea57f45b/volumes" Nov 03 02:42:56.345830 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:42:56.345780 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:01.036520 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:01.036482 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:05.834713 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:05.834674 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:10.410582 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:10.410502 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:13.007794 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:43:13.007755 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:43:15.013717 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:15.013685 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:15.157661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:15.157631 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:15.302280 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:15.302181 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:15.949795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:15.949762 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf"] Nov 03 02:43:16.059065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:16.059028 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79"] Nov 03 02:43:16.327077 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:16.326995 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:16.475636 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:16.475606 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:19.797735 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:19.797687 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:20.148343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:20.148304 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:20.345733 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:20.345683 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:20.950828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:20.950784 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf" podUID="f2251999-6e8a-453b-92d2-c914e4ca59b7" containerName="payload-processing" containerID="cri-o://194fc04144a3588832d877d871dc641c47f95df40eaec61d72c0135e4b466a92" gracePeriod=25 Nov 03 02:43:21.059839 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:21.059794 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79" podUID="98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7" containerName="payload-pre-processing" containerID="cri-o://5f0cb59c0fd1b33583ec62b60e471344e8d12bacb0119d590f0429da5cacbcb2" gracePeriod=25 Nov 03 02:43:21.092150 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:21.092120 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:24.943392 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:24.943342 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:25.506650 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:25.506608 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79" podUID="98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:43:25.506855 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:25.506616 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf" podUID="f2251999-6e8a-453b-92d2-c914e4ca59b7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:43:26.743822 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:26.743783 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:29.639135 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:29.639100 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:34.424340 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:34.424300 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:35.506953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:35.506914 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf" podUID="f2251999-6e8a-453b-92d2-c914e4ca59b7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:43:35.506953 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:35.506948 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79" podUID="98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:43:39.238354 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:39.238315 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:41.266683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.266651 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf" Nov 03 02:43:41.350903 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.350880 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79" Nov 03 02:43:41.412389 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.412357 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2251999-6e8a-453b-92d2-c914e4ca59b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2251999-6e8a-453b-92d2-c914e4ca59b7-plugins-config-volume\") pod \"f2251999-6e8a-453b-92d2-c914e4ca59b7\" (UID: \"f2251999-6e8a-453b-92d2-c914e4ca59b7\") " Nov 03 02:43:41.412578 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.412477 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2251999-6e8a-453b-92d2-c914e4ca59b7-kube-api-access-bjq7z\" (UniqueName: \"kubernetes.io/projected/f2251999-6e8a-453b-92d2-c914e4ca59b7-kube-api-access-bjq7z\") pod \"f2251999-6e8a-453b-92d2-c914e4ca59b7\" (UID: \"f2251999-6e8a-453b-92d2-c914e4ca59b7\") " Nov 03 02:43:41.412765 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.412743 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2251999-6e8a-453b-92d2-c914e4ca59b7-plugins-config-volume" pod "f2251999-6e8a-453b-92d2-c914e4ca59b7" (UID: "f2251999-6e8a-453b-92d2-c914e4ca59b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:43:41.412974 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.412952 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2251999-6e8a-453b-92d2-c914e4ca59b7-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:41.414737 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.414708 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2251999-6e8a-453b-92d2-c914e4ca59b7-kube-api-access-bjq7z" pod "f2251999-6e8a-453b-92d2-c914e4ca59b7" (UID: "f2251999-6e8a-453b-92d2-c914e4ca59b7"). InnerVolumeSpecName "kube-api-access-bjq7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:43:41.432156 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.432131 5463 generic.go:356] "Generic (PLEG): container finished" podID="f2251999-6e8a-453b-92d2-c914e4ca59b7" containerID="194fc04144a3588832d877d871dc641c47f95df40eaec61d72c0135e4b466a92" exitCode=0 Nov 03 02:43:41.432286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.432232 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf" Nov 03 02:43:41.432286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.432236 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf" event={"ID":"f2251999-6e8a-453b-92d2-c914e4ca59b7","Type":"ContainerDied","Data":"194fc04144a3588832d877d871dc641c47f95df40eaec61d72c0135e4b466a92"} Nov 03 02:43:41.432392 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.432300 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf" event={"ID":"f2251999-6e8a-453b-92d2-c914e4ca59b7","Type":"ContainerDied","Data":"fbcd7521aa61d896bc2bb5e52fbaec52d3c291f146b80eca99321017ca84d359"} Nov 03 02:43:41.432392 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.432324 5463 scope.go:122] "RemoveContainer" containerID="194fc04144a3588832d877d871dc641c47f95df40eaec61d72c0135e4b466a92" Nov 03 02:43:41.433846 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.433822 5463 generic.go:356] "Generic (PLEG): container finished" podID="98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7" containerID="5f0cb59c0fd1b33583ec62b60e471344e8d12bacb0119d590f0429da5cacbcb2" exitCode=0 Nov 03 02:43:41.433943 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.433869 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79" Nov 03 02:43:41.433943 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.433875 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79" event={"ID":"98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7","Type":"ContainerDied","Data":"5f0cb59c0fd1b33583ec62b60e471344e8d12bacb0119d590f0429da5cacbcb2"} Nov 03 02:43:41.433943 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.433912 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79" event={"ID":"98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7","Type":"ContainerDied","Data":"2a7ce686a64292c2553df637e14090b5664b9b80d4dc15563cc48ff3465a1e21"} Nov 03 02:43:41.442925 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.442911 5463 scope.go:122] "RemoveContainer" containerID="194fc04144a3588832d877d871dc641c47f95df40eaec61d72c0135e4b466a92" Nov 03 02:43:41.443182 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:43:41.443164 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"194fc04144a3588832d877d871dc641c47f95df40eaec61d72c0135e4b466a92\": container with ID starting with 194fc04144a3588832d877d871dc641c47f95df40eaec61d72c0135e4b466a92 not found: ID does not exist" containerID="194fc04144a3588832d877d871dc641c47f95df40eaec61d72c0135e4b466a92" Nov 03 02:43:41.443280 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.443205 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"194fc04144a3588832d877d871dc641c47f95df40eaec61d72c0135e4b466a92"} err="failed to get container status \"194fc04144a3588832d877d871dc641c47f95df40eaec61d72c0135e4b466a92\": rpc error: code = NotFound desc = could not find container \"194fc04144a3588832d877d871dc641c47f95df40eaec61d72c0135e4b466a92\": container with ID starting with 194fc04144a3588832d877d871dc641c47f95df40eaec61d72c0135e4b466a92 not found: ID does not exist" Nov 03 02:43:41.443280 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.443229 5463 scope.go:122] "RemoveContainer" containerID="5f0cb59c0fd1b33583ec62b60e471344e8d12bacb0119d590f0429da5cacbcb2" Nov 03 02:43:41.452679 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.452657 5463 scope.go:122] "RemoveContainer" containerID="5f0cb59c0fd1b33583ec62b60e471344e8d12bacb0119d590f0429da5cacbcb2" Nov 03 02:43:41.452965 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:43:41.452933 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f0cb59c0fd1b33583ec62b60e471344e8d12bacb0119d590f0429da5cacbcb2\": container with ID starting with 5f0cb59c0fd1b33583ec62b60e471344e8d12bacb0119d590f0429da5cacbcb2 not found: ID does not exist" containerID="5f0cb59c0fd1b33583ec62b60e471344e8d12bacb0119d590f0429da5cacbcb2" Nov 03 02:43:41.453039 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.452974 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f0cb59c0fd1b33583ec62b60e471344e8d12bacb0119d590f0429da5cacbcb2"} err="failed to get container status \"5f0cb59c0fd1b33583ec62b60e471344e8d12bacb0119d590f0429da5cacbcb2\": rpc error: code = NotFound desc = could not find container \"5f0cb59c0fd1b33583ec62b60e471344e8d12bacb0119d590f0429da5cacbcb2\": container with ID starting with 5f0cb59c0fd1b33583ec62b60e471344e8d12bacb0119d590f0429da5cacbcb2 not found: ID does not exist" Nov 03 02:43:41.513914 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.513835 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7-plugins-config-volume\") pod \"98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7\" (UID: \"98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7\") " Nov 03 02:43:41.514087 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.513913 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7-kube-api-access-f2h7h\" (UniqueName: \"kubernetes.io/projected/98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7-kube-api-access-f2h7h\") pod \"98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7\" (UID: \"98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7\") " Nov 03 02:43:41.514250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.514222 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7-plugins-config-volume" pod "98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7" (UID: "98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:43:41.514610 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.514585 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjq7z\" (UniqueName: \"kubernetes.io/projected/f2251999-6e8a-453b-92d2-c914e4ca59b7-kube-api-access-bjq7z\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:41.514610 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.514610 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:41.516689 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.516669 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7-kube-api-access-f2h7h" pod "98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7" (UID: "98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7"). InnerVolumeSpecName "kube-api-access-f2h7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:43:41.611504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.611473 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf"] Nov 03 02:43:41.615364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.615338 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f954db-57bdbb8487-krwpf"] Nov 03 02:43:41.616138 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.616111 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2h7h\" (UniqueName: \"kubernetes.io/projected/98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7-kube-api-access-f2h7h\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:43:41.856306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.856215 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79"] Nov 03 02:43:41.859337 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:41.859309 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f954db-748df585f9-jts79"] Nov 03 02:43:42.444828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:42.444794 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7" path="/var/lib/kubelet/pods/98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7/volumes" Nov 03 02:43:42.445262 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:42.445246 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2251999-6e8a-453b-92d2-c914e4ca59b7" path="/var/lib/kubelet/pods/f2251999-6e8a-453b-92d2-c914e4ca59b7/volumes" Nov 03 02:43:43.009658 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:43:43.009624 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:43:43.922980 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:43.922950 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:49.964928 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:49.964885 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:53.326924 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:53.326887 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:58.031050 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:43:58.031012 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:02.723714 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:02.723677 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:07.562300 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:07.562263 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:12.279483 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:12.279448 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:13.009794 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:44:13.009756 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:44:14.748062 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:14.748016 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:16.990748 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:16.990707 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:20.467725 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:20.467684 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:21.734071 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:21.734033 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:25.862582 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:25.862543 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:27.224307 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:27.224267 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:27.609301 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:27.609221 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:27.814126 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:27.814095 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:27.975686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:27.975647 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:28.227543 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:28.227450 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh"] Nov 03 02:44:28.228034 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:28.227717 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" podUID="cec0591b-d4b8-43f5-a031-6ee4e22c96e5" containerName="istio-proxy" containerID="cri-o://c1ea7b9d886cb1f71785ab174a66963c0c78e9c8684d8e61aa20df683d76dc97" gracePeriod=30 Nov 03 02:44:28.237386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:28.237329 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:28.533313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:28.530338 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:28.966541 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:28.966452 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:29.130341 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:29.130301 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:30.034248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:30.034181 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:30.203798 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:30.203751 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:30.282780 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:30.282742 5463 patch_prober.go:29] interesting pod/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:44:30.282955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:30.282802 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" podUID="cec0591b-d4b8-43f5-a031-6ee4e22c96e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:44:30.360415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:30.360329 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:32.752629 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:32.752573 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:32.927059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:32.927023 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:33.722246 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.721830 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:44:33.786514 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.785780 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-podinfo\") pod \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " Nov 03 02:44:33.786514 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.785882 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-workload-certs\") pod \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " Nov 03 02:44:33.786514 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.785915 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-envoy\") pod \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " Nov 03 02:44:33.787134 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.786852 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-workload-certs" pod "cec0591b-d4b8-43f5-a031-6ee4e22c96e5" (UID: "cec0591b-d4b8-43f5-a031-6ee4e22c96e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:44:33.789826 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.789796 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-envoy" pod "cec0591b-d4b8-43f5-a031-6ee4e22c96e5" (UID: "cec0591b-d4b8-43f5-a031-6ee4e22c96e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:44:33.790308 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.790282 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-podinfo" pod "cec0591b-d4b8-43f5-a031-6ee4e22c96e5" (UID: "cec0591b-d4b8-43f5-a031-6ee4e22c96e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:44:33.887451 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.887405 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-token\" (UniqueName: \"kubernetes.io/projected/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-token\") pod \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " Nov 03 02:44:33.887653 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.887468 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istiod-ca-cert\") pod \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " Nov 03 02:44:33.887653 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.887530 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-credential-socket\") pod \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " Nov 03 02:44:33.887653 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.887573 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-kube-api-access-zpz4p\" (UniqueName: \"kubernetes.io/projected/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-kube-api-access-zpz4p\") pod \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " Nov 03 02:44:33.887851 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.887665 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-data\") pod \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " Nov 03 02:44:33.887851 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.887697 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-workload-socket\") pod \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\" (UID: \"cec0591b-d4b8-43f5-a031-6ee4e22c96e5\") " Nov 03 02:44:33.889381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.888297 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:33.889381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.888326 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:33.889381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.888341 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:33.889381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.888735 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-credential-socket" pod "cec0591b-d4b8-43f5-a031-6ee4e22c96e5" (UID: "cec0591b-d4b8-43f5-a031-6ee4e22c96e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:44:33.889381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.889061 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istiod-ca-cert" pod "cec0591b-d4b8-43f5-a031-6ee4e22c96e5" (UID: "cec0591b-d4b8-43f5-a031-6ee4e22c96e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:44:33.889381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.889404 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-data" pod "cec0591b-d4b8-43f5-a031-6ee4e22c96e5" (UID: "cec0591b-d4b8-43f5-a031-6ee4e22c96e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:44:33.890175 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.890141 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-token" pod "cec0591b-d4b8-43f5-a031-6ee4e22c96e5" (UID: "cec0591b-d4b8-43f5-a031-6ee4e22c96e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:44:33.890325 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.890252 5463 generic.go:356] "Generic (PLEG): container finished" podID="cec0591b-d4b8-43f5-a031-6ee4e22c96e5" containerID="c1ea7b9d886cb1f71785ab174a66963c0c78e9c8684d8e61aa20df683d76dc97" exitCode=0 Nov 03 02:44:33.890391 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.890334 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" event={"ID":"cec0591b-d4b8-43f5-a031-6ee4e22c96e5","Type":"ContainerDied","Data":"c1ea7b9d886cb1f71785ab174a66963c0c78e9c8684d8e61aa20df683d76dc97"} Nov 03 02:44:33.890455 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.890387 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" event={"ID":"cec0591b-d4b8-43f5-a031-6ee4e22c96e5","Type":"ContainerDied","Data":"160964fddcb04a8cd6e2c81ade46d9856fa055eaacca5df3164780f61058d5da"} Nov 03 02:44:33.890455 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.890420 5463 scope.go:122] "RemoveContainer" containerID="c1ea7b9d886cb1f71785ab174a66963c0c78e9c8684d8e61aa20df683d76dc97" Nov 03 02:44:33.890604 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.890587 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh" Nov 03 02:44:33.891358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.891317 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-workload-socket" pod "cec0591b-d4b8-43f5-a031-6ee4e22c96e5" (UID: "cec0591b-d4b8-43f5-a031-6ee4e22c96e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:44:33.891643 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.891618 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-kube-api-access-zpz4p" pod "cec0591b-d4b8-43f5-a031-6ee4e22c96e5" (UID: "cec0591b-d4b8-43f5-a031-6ee4e22c96e5"). InnerVolumeSpecName "kube-api-access-zpz4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:44:33.913828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.913805 5463 scope.go:122] "RemoveContainer" containerID="9597e9b503e48921142c030e540a31efcba4a883220ceb8ee61a4cb3f24081a3" Nov 03 02:44:33.928795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.928772 5463 scope.go:122] "RemoveContainer" containerID="c1ea7b9d886cb1f71785ab174a66963c0c78e9c8684d8e61aa20df683d76dc97" Nov 03 02:44:33.929159 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:44:33.929131 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ea7b9d886cb1f71785ab174a66963c0c78e9c8684d8e61aa20df683d76dc97\": container with ID starting with c1ea7b9d886cb1f71785ab174a66963c0c78e9c8684d8e61aa20df683d76dc97 not found: ID does not exist" containerID="c1ea7b9d886cb1f71785ab174a66963c0c78e9c8684d8e61aa20df683d76dc97" Nov 03 02:44:33.929286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.929173 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ea7b9d886cb1f71785ab174a66963c0c78e9c8684d8e61aa20df683d76dc97"} err="failed to get container status \"c1ea7b9d886cb1f71785ab174a66963c0c78e9c8684d8e61aa20df683d76dc97\": rpc error: code = NotFound desc = could not find container \"c1ea7b9d886cb1f71785ab174a66963c0c78e9c8684d8e61aa20df683d76dc97\": container with ID starting with c1ea7b9d886cb1f71785ab174a66963c0c78e9c8684d8e61aa20df683d76dc97 not found: ID does not exist" Nov 03 02:44:33.929286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.929233 5463 scope.go:122] "RemoveContainer" containerID="9597e9b503e48921142c030e540a31efcba4a883220ceb8ee61a4cb3f24081a3" Nov 03 02:44:33.929549 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:44:33.929518 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9597e9b503e48921142c030e540a31efcba4a883220ceb8ee61a4cb3f24081a3\": container with ID starting with 9597e9b503e48921142c030e540a31efcba4a883220ceb8ee61a4cb3f24081a3 not found: ID does not exist" containerID="9597e9b503e48921142c030e540a31efcba4a883220ceb8ee61a4cb3f24081a3" Nov 03 02:44:33.929673 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.929553 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9597e9b503e48921142c030e540a31efcba4a883220ceb8ee61a4cb3f24081a3"} err="failed to get container status \"9597e9b503e48921142c030e540a31efcba4a883220ceb8ee61a4cb3f24081a3\": rpc error: code = NotFound desc = could not find container \"9597e9b503e48921142c030e540a31efcba4a883220ceb8ee61a4cb3f24081a3\": container with ID starting with 9597e9b503e48921142c030e540a31efcba4a883220ceb8ee61a4cb3f24081a3 not found: ID does not exist" Nov 03 02:44:33.989999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.989954 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:33.989999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.989994 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:33.989999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.990005 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:33.990261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.990014 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpz4p\" (UniqueName: \"kubernetes.io/projected/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-kube-api-access-zpz4p\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:33.990261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.990023 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:33.990261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:33.990032 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec0591b-d4b8-43f5-a031-6ee4e22c96e5-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:34.334260 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:34.324264 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh"] Nov 03 02:44:34.369000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:34.368955 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-f954db-openshift-default-5dd58c47b9-z7kzh"] Nov 03 02:44:34.444386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:34.444349 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cec0591b-d4b8-43f5-a031-6ee4e22c96e5" path="/var/lib/kubelet/pods/cec0591b-d4b8-43f5-a031-6ee4e22c96e5/volumes" Nov 03 02:44:35.401637 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:35.401595 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:36.637357 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:36.637324 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:36.801773 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:36.801734 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:37.294154 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:37.294121 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:39.276582 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:39.276499 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:40.283251 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:40.283208 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:41.285041 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:41.285006 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:41.433785 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:41.433754 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:42.870775 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:42.870725 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:43.012023 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:44:43.011989 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:44:43.333681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:43.333647 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8"] Nov 03 02:44:43.333912 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:43.333886 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" podUID="b64037fe-08de-4969-bdc0-97a3802aa28c" containerName="istio-proxy" containerID="cri-o://574f6ae076f08335342363ef712fc820155bfd1b14bb511aae5faead6570bdb9" gracePeriod=30 Nov 03 02:44:43.361267 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:43.361245 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:44.247322 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:44.247288 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:44.404009 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:44.403981 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:44.954340 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:44.954296 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:45.154504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:45.154466 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:48.621232 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.621205 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:44:48.713275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.713242 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-token\" (UniqueName: \"kubernetes.io/projected/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-token\") pod \"b64037fe-08de-4969-bdc0-97a3802aa28c\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " Nov 03 02:44:48.713479 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.713300 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64037fe-08de-4969-bdc0-97a3802aa28c-kube-api-access-t74r2\" (UniqueName: \"kubernetes.io/projected/b64037fe-08de-4969-bdc0-97a3802aa28c-kube-api-access-t74r2\") pod \"b64037fe-08de-4969-bdc0-97a3802aa28c\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " Nov 03 02:44:48.713479 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.713446 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b64037fe-08de-4969-bdc0-97a3802aa28c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64037fe-08de-4969-bdc0-97a3802aa28c-istiod-ca-cert\") pod \"b64037fe-08de-4969-bdc0-97a3802aa28c\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " Nov 03 02:44:48.713597 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.713482 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-envoy\") pod \"b64037fe-08de-4969-bdc0-97a3802aa28c\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " Nov 03 02:44:48.713653 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.713587 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-podinfo\") pod \"b64037fe-08de-4969-bdc0-97a3802aa28c\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " Nov 03 02:44:48.713653 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.713637 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-workload-socket\") pod \"b64037fe-08de-4969-bdc0-97a3802aa28c\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " Nov 03 02:44:48.713777 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.713730 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-credential-socket\") pod \"b64037fe-08de-4969-bdc0-97a3802aa28c\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " Nov 03 02:44:48.713845 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.713812 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-data\") pod \"b64037fe-08de-4969-bdc0-97a3802aa28c\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " Nov 03 02:44:48.713908 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.713868 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-workload-certs\") pod \"b64037fe-08de-4969-bdc0-97a3802aa28c\" (UID: \"b64037fe-08de-4969-bdc0-97a3802aa28c\") " Nov 03 02:44:48.713996 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.713911 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b64037fe-08de-4969-bdc0-97a3802aa28c-istiod-ca-cert" pod "b64037fe-08de-4969-bdc0-97a3802aa28c" (UID: "b64037fe-08de-4969-bdc0-97a3802aa28c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:44:48.714305 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.714280 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-credential-socket" pod "b64037fe-08de-4969-bdc0-97a3802aa28c" (UID: "b64037fe-08de-4969-bdc0-97a3802aa28c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:44:48.714542 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.714517 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-workload-certs" pod "b64037fe-08de-4969-bdc0-97a3802aa28c" (UID: "b64037fe-08de-4969-bdc0-97a3802aa28c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:44:48.714629 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.714580 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-data" pod "b64037fe-08de-4969-bdc0-97a3802aa28c" (UID: "b64037fe-08de-4969-bdc0-97a3802aa28c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:44:48.714695 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.714630 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-workload-socket" pod "b64037fe-08de-4969-bdc0-97a3802aa28c" (UID: "b64037fe-08de-4969-bdc0-97a3802aa28c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:44:48.714878 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.714847 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64037fe-08de-4969-bdc0-97a3802aa28c-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:48.714957 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.714881 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:48.714957 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.714897 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:48.715613 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.715581 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-token" pod "b64037fe-08de-4969-bdc0-97a3802aa28c" (UID: "b64037fe-08de-4969-bdc0-97a3802aa28c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:44:48.715709 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.715677 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64037fe-08de-4969-bdc0-97a3802aa28c-kube-api-access-t74r2" pod "b64037fe-08de-4969-bdc0-97a3802aa28c" (UID: "b64037fe-08de-4969-bdc0-97a3802aa28c"). InnerVolumeSpecName "kube-api-access-t74r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:44:48.715846 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.715819 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-envoy" pod "b64037fe-08de-4969-bdc0-97a3802aa28c" (UID: "b64037fe-08de-4969-bdc0-97a3802aa28c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:44:48.716493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.716474 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-podinfo" pod "b64037fe-08de-4969-bdc0-97a3802aa28c" (UID: "b64037fe-08de-4969-bdc0-97a3802aa28c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:44:48.816070 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.816035 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:48.816070 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.816068 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t74r2\" (UniqueName: \"kubernetes.io/projected/b64037fe-08de-4969-bdc0-97a3802aa28c-kube-api-access-t74r2\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:48.816294 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.816085 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:48.816294 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.816101 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:48.816294 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.816113 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:48.816294 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:48.816124 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64037fe-08de-4969-bdc0-97a3802aa28c-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:49.064635 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:49.064596 5463 generic.go:356] "Generic (PLEG): container finished" podID="b64037fe-08de-4969-bdc0-97a3802aa28c" containerID="574f6ae076f08335342363ef712fc820155bfd1b14bb511aae5faead6570bdb9" exitCode=0 Nov 03 02:44:49.064872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:49.064653 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" Nov 03 02:44:49.064872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:49.064656 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" event={"ID":"b64037fe-08de-4969-bdc0-97a3802aa28c","Type":"ContainerDied","Data":"574f6ae076f08335342363ef712fc820155bfd1b14bb511aae5faead6570bdb9"} Nov 03 02:44:49.064872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:49.064713 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8" event={"ID":"b64037fe-08de-4969-bdc0-97a3802aa28c","Type":"ContainerDied","Data":"025675925ce4ac652b63676d496b9bdd1331d82d4037810b89dc8aa598e9d723"} Nov 03 02:44:49.064872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:49.064744 5463 scope.go:122] "RemoveContainer" containerID="574f6ae076f08335342363ef712fc820155bfd1b14bb511aae5faead6570bdb9" Nov 03 02:44:49.077163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:49.077140 5463 scope.go:122] "RemoveContainer" containerID="c2c1e3bff8da2b71db8a98253ede81008c60ec15e7607e7047dc2ee9edc3d37b" Nov 03 02:44:49.087540 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:49.087521 5463 scope.go:122] "RemoveContainer" containerID="574f6ae076f08335342363ef712fc820155bfd1b14bb511aae5faead6570bdb9" Nov 03 02:44:49.087825 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:44:49.087809 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574f6ae076f08335342363ef712fc820155bfd1b14bb511aae5faead6570bdb9\": container with ID starting with 574f6ae076f08335342363ef712fc820155bfd1b14bb511aae5faead6570bdb9 not found: ID does not exist" containerID="574f6ae076f08335342363ef712fc820155bfd1b14bb511aae5faead6570bdb9" Nov 03 02:44:49.087875 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:49.087834 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574f6ae076f08335342363ef712fc820155bfd1b14bb511aae5faead6570bdb9"} err="failed to get container status \"574f6ae076f08335342363ef712fc820155bfd1b14bb511aae5faead6570bdb9\": rpc error: code = NotFound desc = could not find container \"574f6ae076f08335342363ef712fc820155bfd1b14bb511aae5faead6570bdb9\": container with ID starting with 574f6ae076f08335342363ef712fc820155bfd1b14bb511aae5faead6570bdb9 not found: ID does not exist" Nov 03 02:44:49.087875 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:49.087850 5463 scope.go:122] "RemoveContainer" containerID="c2c1e3bff8da2b71db8a98253ede81008c60ec15e7607e7047dc2ee9edc3d37b" Nov 03 02:44:49.088123 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:44:49.088109 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c1e3bff8da2b71db8a98253ede81008c60ec15e7607e7047dc2ee9edc3d37b\": container with ID starting with c2c1e3bff8da2b71db8a98253ede81008c60ec15e7607e7047dc2ee9edc3d37b not found: ID does not exist" containerID="c2c1e3bff8da2b71db8a98253ede81008c60ec15e7607e7047dc2ee9edc3d37b" Nov 03 02:44:49.088174 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:49.088130 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c1e3bff8da2b71db8a98253ede81008c60ec15e7607e7047dc2ee9edc3d37b"} err="failed to get container status \"c2c1e3bff8da2b71db8a98253ede81008c60ec15e7607e7047dc2ee9edc3d37b\": rpc error: code = NotFound desc = could not find container \"c2c1e3bff8da2b71db8a98253ede81008c60ec15e7607e7047dc2ee9edc3d37b\": container with ID starting with c2c1e3bff8da2b71db8a98253ede81008c60ec15e7607e7047dc2ee9edc3d37b not found: ID does not exist" Nov 03 02:44:49.231903 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:49.231872 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8"] Nov 03 02:44:49.282361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:49.282330 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-4616ec-openshift-default-6c446c955c-k7wv8"] Nov 03 02:44:49.359750 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:49.359678 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:49.649757 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:49.649729 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:49.835426 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:49.835392 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:49.997219 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:49.997129 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:50.445272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:50.445230 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b64037fe-08de-4969-bdc0-97a3802aa28c" path="/var/lib/kubelet/pods/b64037fe-08de-4969-bdc0-97a3802aa28c/volumes" Nov 03 02:44:52.884312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:52.884275 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:53.053559 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:53.053522 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:54.525407 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:54.525378 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:59.327341 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:59.327296 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:59.830544 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:59.830511 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:59.987140 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:44:59.987098 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:00.163676 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.163641 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm"] Nov 03 02:45:00.164616 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164597 5463 cpu_manager.go:414] "containerMap: removing container" podUID="ba4b9eb7-fc25-451c-9e9f-aae89573754d" containerName="istio-proxy" Nov 03 02:45:00.164682 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164619 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba4b9eb7-fc25-451c-9e9f-aae89573754d" containerName="istio-proxy" Nov 03 02:45:00.164682 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164636 5463 cpu_manager.go:414] "containerMap: removing container" podUID="98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7" containerName="payload-pre-processing" Nov 03 02:45:00.164682 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164644 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7" containerName="payload-pre-processing" Nov 03 02:45:00.164795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164682 5463 cpu_manager.go:414] "containerMap: removing container" podUID="f2251999-6e8a-453b-92d2-c914e4ca59b7" containerName="payload-processing" Nov 03 02:45:00.164795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164690 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2251999-6e8a-453b-92d2-c914e4ca59b7" containerName="payload-processing" Nov 03 02:45:00.164795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164711 5463 cpu_manager.go:414] "containerMap: removing container" podUID="cec0591b-d4b8-43f5-a031-6ee4e22c96e5" containerName="istio-proxy" Nov 03 02:45:00.164795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164718 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec0591b-d4b8-43f5-a031-6ee4e22c96e5" containerName="istio-proxy" Nov 03 02:45:00.164795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164733 5463 cpu_manager.go:414] "containerMap: removing container" podUID="b64037fe-08de-4969-bdc0-97a3802aa28c" containerName="istio-proxy" Nov 03 02:45:00.164795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164740 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64037fe-08de-4969-bdc0-97a3802aa28c" containerName="istio-proxy" Nov 03 02:45:00.164795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164751 5463 cpu_manager.go:414] "containerMap: removing container" podUID="8c87a029-189f-4e51-91dd-4e69ea57f45b" containerName="main" Nov 03 02:45:00.164795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164757 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c87a029-189f-4e51-91dd-4e69ea57f45b" containerName="main" Nov 03 02:45:00.164795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164772 5463 cpu_manager.go:414] "containerMap: removing container" podUID="cec0591b-d4b8-43f5-a031-6ee4e22c96e5" containerName="istio-proxy" Nov 03 02:45:00.164795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164779 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec0591b-d4b8-43f5-a031-6ee4e22c96e5" containerName="istio-proxy" Nov 03 02:45:00.165149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164799 5463 cpu_manager.go:414] "containerMap: removing container" podUID="b64037fe-08de-4969-bdc0-97a3802aa28c" containerName="istio-proxy" Nov 03 02:45:00.165149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164806 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64037fe-08de-4969-bdc0-97a3802aa28c" containerName="istio-proxy" Nov 03 02:45:00.165149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164948 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba4b9eb7-fc25-451c-9e9f-aae89573754d" containerName="istio-proxy" Nov 03 02:45:00.165149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164960 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a4bbfb-fb5a-44f1-8cbe-bf808d4b1db7" containerName="payload-pre-processing" Nov 03 02:45:00.165149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164976 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2251999-6e8a-453b-92d2-c914e4ca59b7" containerName="payload-processing" Nov 03 02:45:00.165149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164987 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec0591b-d4b8-43f5-a031-6ee4e22c96e5" containerName="istio-proxy" Nov 03 02:45:00.165149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.164995 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64037fe-08de-4969-bdc0-97a3802aa28c" containerName="istio-proxy" Nov 03 02:45:00.165149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.165004 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c87a029-189f-4e51-91dd-4e69ea57f45b" containerName="main" Nov 03 02:45:00.165149 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.165017 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64037fe-08de-4969-bdc0-97a3802aa28c" containerName="istio-proxy" Nov 03 02:45:00.169112 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.169090 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm" Nov 03 02:45:00.184418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.184395 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq"] Nov 03 02:45:00.185469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.185442 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec0591b-d4b8-43f5-a031-6ee4e22c96e5" containerName="istio-proxy" Nov 03 02:45:00.189343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.189325 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq" Nov 03 02:45:00.190344 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.190269 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc"] Nov 03 02:45:00.195175 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.195154 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" Nov 03 02:45:00.213423 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.213392 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52"] Nov 03 02:45:00.217417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.217403 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm"] Nov 03 02:45:00.217526 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.217510 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" Nov 03 02:45:00.221137 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.221116 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" Nov 03 02:45:00.234586 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.234560 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8"] Nov 03 02:45:00.245439 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.245408 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" Nov 03 02:45:00.275881 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.275854 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02969c8-4434-49f0-8d59-b45008195cbf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm\" (UID: \"d02969c8-4434-49f0-8d59-b45008195cbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm" Nov 03 02:45:00.276025 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.275894 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da74f893-34d6-43bf-b814-83f4e2cab7ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894565-c4j52\" (UID: \"da74f893-34d6-43bf-b814-83f4e2cab7ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" Nov 03 02:45:00.276025 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.275917 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjth8\" (UniqueName: \"kubernetes.io/projected/da74f893-34d6-43bf-b814-83f4e2cab7ac-kube-api-access-hjth8\") pod \"maas-api-key-cleanup-29894565-c4j52\" (UID: \"da74f893-34d6-43bf-b814-83f4e2cab7ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" Nov 03 02:45:00.276121 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.276042 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8958w\" (UniqueName: \"kubernetes.io/projected/a70245c9-bbc6-4be9-b59a-783f65e3dae8-kube-api-access-8958w\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc\" (UID: \"a70245c9-bbc6-4be9-b59a-783f65e3dae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" Nov 03 02:45:00.276121 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.276072 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr2mz\" (UniqueName: \"kubernetes.io/projected/e47bcd1a-f924-497d-8cb1-eaf0839d354f-kube-api-access-fr2mz\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq\" (UID: \"e47bcd1a-f924-497d-8cb1-eaf0839d354f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq" Nov 03 02:45:00.276231 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.276182 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70245c9-bbc6-4be9-b59a-783f65e3dae8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc\" (UID: \"a70245c9-bbc6-4be9-b59a-783f65e3dae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" Nov 03 02:45:00.276304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.276273 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a5260c-8fdb-4b04-b48c-0d148fc8612b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm\" (UID: \"42a5260c-8fdb-4b04-b48c-0d148fc8612b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" Nov 03 02:45:00.276366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.276334 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d5e25d2-cf73-44b8-82f8-5789bedcc9e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8\" (UID: \"3d5e25d2-cf73-44b8-82f8-5789bedcc9e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" Nov 03 02:45:00.276431 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.276417 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhtcf\" (UniqueName: \"kubernetes.io/projected/d02969c8-4434-49f0-8d59-b45008195cbf-kube-api-access-vhtcf\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm\" (UID: \"d02969c8-4434-49f0-8d59-b45008195cbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm" Nov 03 02:45:00.276524 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.276510 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whxqh\" (UniqueName: \"kubernetes.io/projected/3d5e25d2-cf73-44b8-82f8-5789bedcc9e1-kube-api-access-whxqh\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8\" (UID: \"3d5e25d2-cf73-44b8-82f8-5789bedcc9e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" Nov 03 02:45:00.276620 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.276600 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgbj5\" (UniqueName: \"kubernetes.io/projected/42a5260c-8fdb-4b04-b48c-0d148fc8612b-kube-api-access-xgbj5\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm\" (UID: \"42a5260c-8fdb-4b04-b48c-0d148fc8612b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" Nov 03 02:45:00.276702 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.276685 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e47bcd1a-f924-497d-8cb1-eaf0839d354f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq\" (UID: \"e47bcd1a-f924-497d-8cb1-eaf0839d354f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq" Nov 03 02:45:00.298176 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.298149 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm"] Nov 03 02:45:00.310907 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.310870 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq"] Nov 03 02:45:00.311077 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.310932 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc"] Nov 03 02:45:00.315248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.315221 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52"] Nov 03 02:45:00.315598 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.315576 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 02:45:00.348949 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.348921 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm"] Nov 03 02:45:00.375748 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.375665 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8"] Nov 03 02:45:00.377833 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.377800 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whxqh\" (UniqueName: \"kubernetes.io/projected/3d5e25d2-cf73-44b8-82f8-5789bedcc9e1-kube-api-access-whxqh\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8\" (UID: \"3d5e25d2-cf73-44b8-82f8-5789bedcc9e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" Nov 03 02:45:00.377983 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.377888 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgbj5\" (UniqueName: \"kubernetes.io/projected/42a5260c-8fdb-4b04-b48c-0d148fc8612b-kube-api-access-xgbj5\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm\" (UID: \"42a5260c-8fdb-4b04-b48c-0d148fc8612b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" Nov 03 02:45:00.377983 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.377943 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e47bcd1a-f924-497d-8cb1-eaf0839d354f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq\" (UID: \"e47bcd1a-f924-497d-8cb1-eaf0839d354f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq" Nov 03 02:45:00.378106 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.378018 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02969c8-4434-49f0-8d59-b45008195cbf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm\" (UID: \"d02969c8-4434-49f0-8d59-b45008195cbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm" Nov 03 02:45:00.378106 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.378051 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da74f893-34d6-43bf-b814-83f4e2cab7ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894565-c4j52\" (UID: \"da74f893-34d6-43bf-b814-83f4e2cab7ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" Nov 03 02:45:00.378106 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.378077 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjth8\" (UniqueName: \"kubernetes.io/projected/da74f893-34d6-43bf-b814-83f4e2cab7ac-kube-api-access-hjth8\") pod \"maas-api-key-cleanup-29894565-c4j52\" (UID: \"da74f893-34d6-43bf-b814-83f4e2cab7ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" Nov 03 02:45:00.378361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.378117 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8958w\" (UniqueName: \"kubernetes.io/projected/a70245c9-bbc6-4be9-b59a-783f65e3dae8-kube-api-access-8958w\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc\" (UID: \"a70245c9-bbc6-4be9-b59a-783f65e3dae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" Nov 03 02:45:00.378361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.378157 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr2mz\" (UniqueName: \"kubernetes.io/projected/e47bcd1a-f924-497d-8cb1-eaf0839d354f-kube-api-access-fr2mz\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq\" (UID: \"e47bcd1a-f924-497d-8cb1-eaf0839d354f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq" Nov 03 02:45:00.378361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.378261 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70245c9-bbc6-4be9-b59a-783f65e3dae8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc\" (UID: \"a70245c9-bbc6-4be9-b59a-783f65e3dae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" Nov 03 02:45:00.378361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.378310 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a5260c-8fdb-4b04-b48c-0d148fc8612b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm\" (UID: \"42a5260c-8fdb-4b04-b48c-0d148fc8612b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" Nov 03 02:45:00.378361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.378345 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d5e25d2-cf73-44b8-82f8-5789bedcc9e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8\" (UID: \"3d5e25d2-cf73-44b8-82f8-5789bedcc9e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" Nov 03 02:45:00.378655 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.378400 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhtcf\" (UniqueName: \"kubernetes.io/projected/d02969c8-4434-49f0-8d59-b45008195cbf-kube-api-access-vhtcf\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm\" (UID: \"d02969c8-4434-49f0-8d59-b45008195cbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm" Nov 03 02:45:00.379967 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.379941 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e47bcd1a-f924-497d-8cb1-eaf0839d354f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq\" (UID: \"e47bcd1a-f924-497d-8cb1-eaf0839d354f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq" Nov 03 02:45:00.380710 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.380686 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02969c8-4434-49f0-8d59-b45008195cbf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm\" (UID: \"d02969c8-4434-49f0-8d59-b45008195cbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm" Nov 03 02:45:00.381387 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.381364 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da74f893-34d6-43bf-b814-83f4e2cab7ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894565-c4j52\" (UID: \"da74f893-34d6-43bf-b814-83f4e2cab7ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" Nov 03 02:45:00.382769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.382743 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70245c9-bbc6-4be9-b59a-783f65e3dae8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc\" (UID: \"a70245c9-bbc6-4be9-b59a-783f65e3dae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" Nov 03 02:45:00.383641 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.383614 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d5e25d2-cf73-44b8-82f8-5789bedcc9e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8\" (UID: \"3d5e25d2-cf73-44b8-82f8-5789bedcc9e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" Nov 03 02:45:00.384749 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.384723 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a5260c-8fdb-4b04-b48c-0d148fc8612b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm\" (UID: \"42a5260c-8fdb-4b04-b48c-0d148fc8612b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" Nov 03 02:45:00.436485 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.436413 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whxqh\" (UniqueName: \"kubernetes.io/projected/3d5e25d2-cf73-44b8-82f8-5789bedcc9e1-kube-api-access-whxqh\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8\" (UID: \"3d5e25d2-cf73-44b8-82f8-5789bedcc9e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" Nov 03 02:45:00.438103 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.438074 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgbj5\" (UniqueName: \"kubernetes.io/projected/42a5260c-8fdb-4b04-b48c-0d148fc8612b-kube-api-access-xgbj5\") pod \"maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm\" (UID: \"42a5260c-8fdb-4b04-b48c-0d148fc8612b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" Nov 03 02:45:00.438312 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.438292 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhtcf\" (UniqueName: \"kubernetes.io/projected/d02969c8-4434-49f0-8d59-b45008195cbf-kube-api-access-vhtcf\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm\" (UID: \"d02969c8-4434-49f0-8d59-b45008195cbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm" Nov 03 02:45:00.438714 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.438695 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjth8\" (UniqueName: \"kubernetes.io/projected/da74f893-34d6-43bf-b814-83f4e2cab7ac-kube-api-access-hjth8\") pod \"maas-api-key-cleanup-29894565-c4j52\" (UID: \"da74f893-34d6-43bf-b814-83f4e2cab7ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" Nov 03 02:45:00.439099 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.439076 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8958w\" (UniqueName: \"kubernetes.io/projected/a70245c9-bbc6-4be9-b59a-783f65e3dae8-kube-api-access-8958w\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc\" (UID: \"a70245c9-bbc6-4be9-b59a-783f65e3dae8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" Nov 03 02:45:00.439645 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.439627 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr2mz\" (UniqueName: \"kubernetes.io/projected/e47bcd1a-f924-497d-8cb1-eaf0839d354f-kube-api-access-fr2mz\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq\" (UID: \"e47bcd1a-f924-497d-8cb1-eaf0839d354f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq" Nov 03 02:45:00.484825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.484793 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm" Nov 03 02:45:00.507293 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.507266 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq" Nov 03 02:45:00.512547 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.512516 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" Nov 03 02:45:00.531739 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.531710 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" Nov 03 02:45:00.547139 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.546906 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" Nov 03 02:45:00.572007 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.564400 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" Nov 03 02:45:00.878738 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.872987 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm"] Nov 03 02:45:00.931552 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.931524 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq"] Nov 03 02:45:00.931951 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:45:00.931926 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47bcd1a_f924_497d_8cb1_eaf0839d354f.slice/crio-5239b51d1bd35166ad9ff7b4cdc751d14a3a927d744039440ec0e6162c601e42 WatchSource:0}: Error finding container 5239b51d1bd35166ad9ff7b4cdc751d14a3a927d744039440ec0e6162c601e42: Status 404 returned error can't find the container with id 5239b51d1bd35166ad9ff7b4cdc751d14a3a927d744039440ec0e6162c601e42 Nov 03 02:45:00.962631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:00.962361 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52"] Nov 03 02:45:01.018857 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:01.018832 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm"] Nov 03 02:45:01.022364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:01.022340 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc"] Nov 03 02:45:01.041294 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:01.041271 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8"] Nov 03 02:45:01.050290 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:45:01.050260 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a5260c_8fdb_4b04_b48c_0d148fc8612b.slice/crio-ad72551ee703076b8bca34c932558ecd22d522401ccd0e7fdee5b0b70d37befa WatchSource:0}: Error finding container ad72551ee703076b8bca34c932558ecd22d522401ccd0e7fdee5b0b70d37befa: Status 404 returned error can't find the container with id ad72551ee703076b8bca34c932558ecd22d522401ccd0e7fdee5b0b70d37befa Nov 03 02:45:01.051886 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:45:01.051852 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70245c9_bbc6_4be9_b59a_783f65e3dae8.slice/crio-510d9cf97866b420d2c5e9f560570cc9dd00d50bda7094fb654395eec9da816c WatchSource:0}: Error finding container 510d9cf97866b420d2c5e9f560570cc9dd00d50bda7094fb654395eec9da816c: Status 404 returned error can't find the container with id 510d9cf97866b420d2c5e9f560570cc9dd00d50bda7094fb654395eec9da816c Nov 03 02:45:01.053305 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:45:01.053275 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5e25d2_cf73_44b8_82f8_5789bedcc9e1.slice/crio-91ab49d88b4e28ec9c125ac6fa8f5f024b1caa0dca08b3184881322eaf7b9e10 WatchSource:0}: Error finding container 91ab49d88b4e28ec9c125ac6fa8f5f024b1caa0dca08b3184881322eaf7b9e10: Status 404 returned error can't find the container with id 91ab49d88b4e28ec9c125ac6fa8f5f024b1caa0dca08b3184881322eaf7b9e10 Nov 03 02:45:01.164467 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:01.164434 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" event={"ID":"da74f893-34d6-43bf-b814-83f4e2cab7ac","Type":"ContainerStarted","Data":"23f95ab2304c1ccfc5f543498769f16e5b2e57addef039028970cce3eaf24d43"} Nov 03 02:45:01.165929 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:01.165899 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm" event={"ID":"d02969c8-4434-49f0-8d59-b45008195cbf","Type":"ContainerStarted","Data":"cb3391cc1f91576e3ae119ba5b2f08bfbfb0604a4169b1ce18b66a0281104e09"} Nov 03 02:45:01.167343 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:01.167322 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" event={"ID":"a70245c9-bbc6-4be9-b59a-783f65e3dae8","Type":"ContainerStarted","Data":"510d9cf97866b420d2c5e9f560570cc9dd00d50bda7094fb654395eec9da816c"} Nov 03 02:45:01.168769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:01.168745 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq" event={"ID":"e47bcd1a-f924-497d-8cb1-eaf0839d354f","Type":"ContainerStarted","Data":"5239b51d1bd35166ad9ff7b4cdc751d14a3a927d744039440ec0e6162c601e42"} Nov 03 02:45:01.170090 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:01.170065 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" event={"ID":"42a5260c-8fdb-4b04-b48c-0d148fc8612b","Type":"ContainerStarted","Data":"ad72551ee703076b8bca34c932558ecd22d522401ccd0e7fdee5b0b70d37befa"} Nov 03 02:45:01.171480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:01.171452 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" event={"ID":"3d5e25d2-cf73-44b8-82f8-5789bedcc9e1","Type":"ContainerStarted","Data":"91ab49d88b4e28ec9c125ac6fa8f5f024b1caa0dca08b3184881322eaf7b9e10"} Nov 03 02:45:02.181302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:02.181268 5463 generic.go:356] "Generic (PLEG): container finished" podID="3d5e25d2-cf73-44b8-82f8-5789bedcc9e1" containerID="fcada50d06fa45a04c3a188a15542cc3a6595ab54b8427fe6e0fd15204e48f17" exitCode=60 Nov 03 02:45:02.181701 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:02.181353 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" event={"ID":"3d5e25d2-cf73-44b8-82f8-5789bedcc9e1","Type":"ContainerDied","Data":"fcada50d06fa45a04c3a188a15542cc3a6595ab54b8427fe6e0fd15204e48f17"} Nov 03 02:45:02.181701 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:02.181642 5463 scope.go:122] "RemoveContainer" containerID="fcada50d06fa45a04c3a188a15542cc3a6595ab54b8427fe6e0fd15204e48f17" Nov 03 02:45:02.183244 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:02.183181 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" event={"ID":"da74f893-34d6-43bf-b814-83f4e2cab7ac","Type":"ContainerStarted","Data":"053ada731e87c1bb8699c6c48a3b545a7bdcca136908d8851052d80632aa1db3"} Nov 03 02:45:02.185361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:02.185307 5463 generic.go:356] "Generic (PLEG): container finished" podID="d02969c8-4434-49f0-8d59-b45008195cbf" containerID="6141b1bc710054cafc0042bcc946c61d7d14c0635bd6f82b2a2e35ea2b8eaa14" exitCode=60 Nov 03 02:45:02.185434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:02.185404 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm" event={"ID":"d02969c8-4434-49f0-8d59-b45008195cbf","Type":"ContainerDied","Data":"6141b1bc710054cafc0042bcc946c61d7d14c0635bd6f82b2a2e35ea2b8eaa14"} Nov 03 02:45:02.185634 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:02.185615 5463 scope.go:122] "RemoveContainer" containerID="6141b1bc710054cafc0042bcc946c61d7d14c0635bd6f82b2a2e35ea2b8eaa14" Nov 03 02:45:02.187880 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:02.187856 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" event={"ID":"a70245c9-bbc6-4be9-b59a-783f65e3dae8","Type":"ContainerStarted","Data":"34889b17827fdf414e07fd297cdb32c67d202b31dc9a6f3b7d8f798692046f21"} Nov 03 02:45:02.196284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:02.190299 5463 generic.go:356] "Generic (PLEG): container finished" podID="e47bcd1a-f924-497d-8cb1-eaf0839d354f" containerID="371004ad78305a08e5ad167662d1a050d1db1202bad7455f5de9232858c1a951" exitCode=60 Nov 03 02:45:02.196284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:02.190378 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq" event={"ID":"e47bcd1a-f924-497d-8cb1-eaf0839d354f","Type":"ContainerDied","Data":"371004ad78305a08e5ad167662d1a050d1db1202bad7455f5de9232858c1a951"} Nov 03 02:45:02.196284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:02.190587 5463 scope.go:122] "RemoveContainer" containerID="371004ad78305a08e5ad167662d1a050d1db1202bad7455f5de9232858c1a951" Nov 03 02:45:02.196284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:02.192117 5463 generic.go:356] "Generic (PLEG): container finished" podID="42a5260c-8fdb-4b04-b48c-0d148fc8612b" containerID="bbad4504437550acd3dda2df035295730a79203f9068b99de3f289b7e311d952" exitCode=60 Nov 03 02:45:02.196284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:02.192292 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" event={"ID":"42a5260c-8fdb-4b04-b48c-0d148fc8612b","Type":"ContainerDied","Data":"bbad4504437550acd3dda2df035295730a79203f9068b99de3f289b7e311d952"} Nov 03 02:45:02.196284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:02.192647 5463 scope.go:122] "RemoveContainer" containerID="bbad4504437550acd3dda2df035295730a79203f9068b99de3f289b7e311d952" Nov 03 02:45:02.409733 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:02.409684 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" podStartSLOduration=2.409670819 podStartE2EDuration="2.409670819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:02.408905106 +0000 UTC m=+3184.093014263" watchObservedRunningTime="2026-11-03 02:45:02.409670819 +0000 UTC m=+3184.093780031" Nov 03 02:45:02.525176 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:02.525119 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" podStartSLOduration=2.525101989 podStartE2EDuration="2.525101989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:02.522130514 +0000 UTC m=+3184.206239671" watchObservedRunningTime="2026-11-03 02:45:02.525101989 +0000 UTC m=+3184.209211146" Nov 03 02:45:03.201539 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:03.201506 5463 generic.go:356] "Generic (PLEG): container finished" podID="da74f893-34d6-43bf-b814-83f4e2cab7ac" containerID="053ada731e87c1bb8699c6c48a3b545a7bdcca136908d8851052d80632aa1db3" exitCode=60 Nov 03 02:45:03.202029 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:03.201552 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" event={"ID":"da74f893-34d6-43bf-b814-83f4e2cab7ac","Type":"ContainerDied","Data":"053ada731e87c1bb8699c6c48a3b545a7bdcca136908d8851052d80632aa1db3"} Nov 03 02:45:03.202029 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:03.201965 5463 scope.go:122] "RemoveContainer" containerID="053ada731e87c1bb8699c6c48a3b545a7bdcca136908d8851052d80632aa1db3" Nov 03 02:45:03.203769 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:03.203730 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm" event={"ID":"d02969c8-4434-49f0-8d59-b45008195cbf","Type":"ContainerStarted","Data":"7cac496cc64d2167a0cfdfc0ff6af9b8301e00906653ecd3656a3b489d62b591"} Nov 03 02:45:03.205325 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:03.205303 5463 generic.go:356] "Generic (PLEG): container finished" podID="a70245c9-bbc6-4be9-b59a-783f65e3dae8" containerID="34889b17827fdf414e07fd297cdb32c67d202b31dc9a6f3b7d8f798692046f21" exitCode=60 Nov 03 02:45:03.205435 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:03.205373 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" event={"ID":"a70245c9-bbc6-4be9-b59a-783f65e3dae8","Type":"ContainerDied","Data":"34889b17827fdf414e07fd297cdb32c67d202b31dc9a6f3b7d8f798692046f21"} Nov 03 02:45:03.205631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:03.205614 5463 scope.go:122] "RemoveContainer" containerID="34889b17827fdf414e07fd297cdb32c67d202b31dc9a6f3b7d8f798692046f21" Nov 03 02:45:03.207384 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:03.207349 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq" event={"ID":"e47bcd1a-f924-497d-8cb1-eaf0839d354f","Type":"ContainerStarted","Data":"648d29f72c95708831e1f40bf350b3316ae903c3bad48891be5511fb7619c392"} Nov 03 02:45:03.209406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:03.209378 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" event={"ID":"42a5260c-8fdb-4b04-b48c-0d148fc8612b","Type":"ContainerStarted","Data":"4ba82e231249f1c0cc31d5a2e2a6023316a3931d53fd15bc7c0d648aa0a3c4d2"} Nov 03 02:45:03.211559 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:03.211535 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" event={"ID":"3d5e25d2-cf73-44b8-82f8-5789bedcc9e1","Type":"ContainerStarted","Data":"53753c0747a1e4c10507e41fcd997bea655be82caca7987af4f988b7a3909f76"} Nov 03 02:45:03.429790 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:03.429734 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm" podStartSLOduration=3.429720952 podStartE2EDuration="3.429720952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:03.427651398 +0000 UTC m=+3185.111760553" watchObservedRunningTime="2026-11-03 02:45:03.429720952 +0000 UTC m=+3185.113830107" Nov 03 02:45:03.659409 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:03.659363 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq" podStartSLOduration=3.659348516 podStartE2EDuration="3.659348516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:03.655366909 +0000 UTC m=+3185.339476064" watchObservedRunningTime="2026-11-03 02:45:03.659348516 +0000 UTC m=+3185.343457670" Nov 03 02:45:03.769902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:03.769855 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" podStartSLOduration=3.769841993 podStartE2EDuration="3.769841993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:03.768252502 +0000 UTC m=+3185.452361656" watchObservedRunningTime="2026-11-03 02:45:03.769841993 +0000 UTC m=+3185.453951147" Nov 03 02:45:03.883050 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:03.882996 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" podStartSLOduration=3.882978536 podStartE2EDuration="3.882978536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:03.882230514 +0000 UTC m=+3185.566339672" watchObservedRunningTime="2026-11-03 02:45:03.882978536 +0000 UTC m=+3185.567087693" Nov 03 02:45:04.129127 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:04.129000 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:04.221050 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:04.221013 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" event={"ID":"da74f893-34d6-43bf-b814-83f4e2cab7ac","Type":"ContainerStarted","Data":"112f7e7505f3d6f14d8c2318ee78ba930da59f79fabb18a4bdeee2bb1783c37c"} Nov 03 02:45:04.223467 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:04.223442 5463 generic.go:356] "Generic (PLEG): container finished" podID="d02969c8-4434-49f0-8d59-b45008195cbf" containerID="7cac496cc64d2167a0cfdfc0ff6af9b8301e00906653ecd3656a3b489d62b591" exitCode=60 Nov 03 02:45:04.223599 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:04.223478 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm" event={"ID":"d02969c8-4434-49f0-8d59-b45008195cbf","Type":"ContainerDied","Data":"7cac496cc64d2167a0cfdfc0ff6af9b8301e00906653ecd3656a3b489d62b591"} Nov 03 02:45:04.223599 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:04.223536 5463 scope.go:122] "RemoveContainer" containerID="6141b1bc710054cafc0042bcc946c61d7d14c0635bd6f82b2a2e35ea2b8eaa14" Nov 03 02:45:04.223844 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:04.223829 5463 scope.go:122] "RemoveContainer" containerID="7cac496cc64d2167a0cfdfc0ff6af9b8301e00906653ecd3656a3b489d62b591" Nov 03 02:45:04.224171 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:04.224149 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm_odh-ai-gateway-infra(d02969c8-4434-49f0-8d59-b45008195cbf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm" podUID="d02969c8-4434-49f0-8d59-b45008195cbf" Nov 03 02:45:04.225445 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:04.225421 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" event={"ID":"a70245c9-bbc6-4be9-b59a-783f65e3dae8","Type":"ContainerStarted","Data":"2834549cc34c33417951105cb910246fa23922034708de32eb7014402f67b631"} Nov 03 02:45:04.227370 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:04.227351 5463 generic.go:356] "Generic (PLEG): container finished" podID="e47bcd1a-f924-497d-8cb1-eaf0839d354f" containerID="648d29f72c95708831e1f40bf350b3316ae903c3bad48891be5511fb7619c392" exitCode=60 Nov 03 02:45:04.227477 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:04.227430 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq" event={"ID":"e47bcd1a-f924-497d-8cb1-eaf0839d354f","Type":"ContainerDied","Data":"648d29f72c95708831e1f40bf350b3316ae903c3bad48891be5511fb7619c392"} Nov 03 02:45:04.227612 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:04.227598 5463 scope.go:122] "RemoveContainer" containerID="648d29f72c95708831e1f40bf350b3316ae903c3bad48891be5511fb7619c392" Nov 03 02:45:04.227861 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:04.227839 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq_odh-ai-gateway-infra(e47bcd1a-f924-497d-8cb1-eaf0839d354f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq" podUID="e47bcd1a-f924-497d-8cb1-eaf0839d354f" Nov 03 02:45:04.229350 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:04.229327 5463 generic.go:356] "Generic (PLEG): container finished" podID="42a5260c-8fdb-4b04-b48c-0d148fc8612b" containerID="4ba82e231249f1c0cc31d5a2e2a6023316a3931d53fd15bc7c0d648aa0a3c4d2" exitCode=60 Nov 03 02:45:04.229441 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:04.229404 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" event={"ID":"42a5260c-8fdb-4b04-b48c-0d148fc8612b","Type":"ContainerDied","Data":"4ba82e231249f1c0cc31d5a2e2a6023316a3931d53fd15bc7c0d648aa0a3c4d2"} Nov 03 02:45:04.229567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:04.229547 5463 scope.go:122] "RemoveContainer" containerID="4ba82e231249f1c0cc31d5a2e2a6023316a3931d53fd15bc7c0d648aa0a3c4d2" Nov 03 02:45:04.229764 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:04.229744 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm_odh-ai-gateway-infra(42a5260c-8fdb-4b04-b48c-0d148fc8612b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" podUID="42a5260c-8fdb-4b04-b48c-0d148fc8612b" Nov 03 02:45:04.231391 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:04.231373 5463 generic.go:356] "Generic (PLEG): container finished" podID="3d5e25d2-cf73-44b8-82f8-5789bedcc9e1" containerID="53753c0747a1e4c10507e41fcd997bea655be82caca7987af4f988b7a3909f76" exitCode=60 Nov 03 02:45:04.231470 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:04.231424 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" event={"ID":"3d5e25d2-cf73-44b8-82f8-5789bedcc9e1","Type":"ContainerDied","Data":"53753c0747a1e4c10507e41fcd997bea655be82caca7987af4f988b7a3909f76"} Nov 03 02:45:04.231627 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:04.231612 5463 scope.go:122] "RemoveContainer" containerID="53753c0747a1e4c10507e41fcd997bea655be82caca7987af4f988b7a3909f76" Nov 03 02:45:04.231818 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:04.231802 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8_odh-ai-gateway-infra(3d5e25d2-cf73-44b8-82f8-5789bedcc9e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" podUID="3d5e25d2-cf73-44b8-82f8-5789bedcc9e1" Nov 03 02:45:04.237046 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:04.237029 5463 scope.go:122] "RemoveContainer" containerID="371004ad78305a08e5ad167662d1a050d1db1202bad7455f5de9232858c1a951" Nov 03 02:45:04.251003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:04.250980 5463 scope.go:122] "RemoveContainer" containerID="bbad4504437550acd3dda2df035295730a79203f9068b99de3f289b7e311d952" Nov 03 02:45:04.314545 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:04.314523 5463 scope.go:122] "RemoveContainer" containerID="fcada50d06fa45a04c3a188a15542cc3a6595ab54b8427fe6e0fd15204e48f17" Nov 03 02:45:05.241250 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:05.241224 5463 scope.go:122] "RemoveContainer" containerID="53753c0747a1e4c10507e41fcd997bea655be82caca7987af4f988b7a3909f76" Nov 03 02:45:05.241754 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:05.241557 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8_odh-ai-gateway-infra(3d5e25d2-cf73-44b8-82f8-5789bedcc9e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" podUID="3d5e25d2-cf73-44b8-82f8-5789bedcc9e1" Nov 03 02:45:05.242854 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:05.242824 5463 generic.go:356] "Generic (PLEG): container finished" podID="da74f893-34d6-43bf-b814-83f4e2cab7ac" containerID="112f7e7505f3d6f14d8c2318ee78ba930da59f79fabb18a4bdeee2bb1783c37c" exitCode=60 Nov 03 02:45:05.243012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:05.242907 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" event={"ID":"da74f893-34d6-43bf-b814-83f4e2cab7ac","Type":"ContainerDied","Data":"112f7e7505f3d6f14d8c2318ee78ba930da59f79fabb18a4bdeee2bb1783c37c"} Nov 03 02:45:05.243012 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:05.242963 5463 scope.go:122] "RemoveContainer" containerID="053ada731e87c1bb8699c6c48a3b545a7bdcca136908d8851052d80632aa1db3" Nov 03 02:45:05.243276 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:05.243256 5463 scope.go:122] "RemoveContainer" containerID="112f7e7505f3d6f14d8c2318ee78ba930da59f79fabb18a4bdeee2bb1783c37c" Nov 03 02:45:05.243636 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:05.243600 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894565-c4j52_odh-ai-gateway-infra(da74f893-34d6-43bf-b814-83f4e2cab7ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" podUID="da74f893-34d6-43bf-b814-83f4e2cab7ac" Nov 03 02:45:05.245239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:05.245219 5463 scope.go:122] "RemoveContainer" containerID="7cac496cc64d2167a0cfdfc0ff6af9b8301e00906653ecd3656a3b489d62b591" Nov 03 02:45:05.245515 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:05.245497 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm_odh-ai-gateway-infra(d02969c8-4434-49f0-8d59-b45008195cbf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm" podUID="d02969c8-4434-49f0-8d59-b45008195cbf" Nov 03 02:45:05.247309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:05.247288 5463 generic.go:356] "Generic (PLEG): container finished" podID="a70245c9-bbc6-4be9-b59a-783f65e3dae8" containerID="2834549cc34c33417951105cb910246fa23922034708de32eb7014402f67b631" exitCode=60 Nov 03 02:45:05.247417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:05.247369 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" event={"ID":"a70245c9-bbc6-4be9-b59a-783f65e3dae8","Type":"ContainerDied","Data":"2834549cc34c33417951105cb910246fa23922034708de32eb7014402f67b631"} Nov 03 02:45:05.247647 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:05.247627 5463 scope.go:122] "RemoveContainer" containerID="2834549cc34c33417951105cb910246fa23922034708de32eb7014402f67b631" Nov 03 02:45:05.247917 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:05.247894 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc_odh-ai-gateway-infra(a70245c9-bbc6-4be9-b59a-783f65e3dae8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" podUID="a70245c9-bbc6-4be9-b59a-783f65e3dae8" Nov 03 02:45:05.249657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:05.249637 5463 scope.go:122] "RemoveContainer" containerID="648d29f72c95708831e1f40bf350b3316ae903c3bad48891be5511fb7619c392" Nov 03 02:45:05.250022 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:05.249835 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq_odh-ai-gateway-infra(e47bcd1a-f924-497d-8cb1-eaf0839d354f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq" podUID="e47bcd1a-f924-497d-8cb1-eaf0839d354f" Nov 03 02:45:05.252038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:05.252023 5463 scope.go:122] "RemoveContainer" containerID="4ba82e231249f1c0cc31d5a2e2a6023316a3931d53fd15bc7c0d648aa0a3c4d2" Nov 03 02:45:05.252266 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:05.252241 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm_odh-ai-gateway-infra(42a5260c-8fdb-4b04-b48c-0d148fc8612b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" podUID="42a5260c-8fdb-4b04-b48c-0d148fc8612b" Nov 03 02:45:05.260690 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:05.260674 5463 scope.go:122] "RemoveContainer" containerID="34889b17827fdf414e07fd297cdb32c67d202b31dc9a6f3b7d8f798692046f21" Nov 03 02:45:06.262571 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:06.262544 5463 scope.go:122] "RemoveContainer" containerID="112f7e7505f3d6f14d8c2318ee78ba930da59f79fabb18a4bdeee2bb1783c37c" Nov 03 02:45:06.263140 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:06.262860 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894565-c4j52_odh-ai-gateway-infra(da74f893-34d6-43bf-b814-83f4e2cab7ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" podUID="da74f893-34d6-43bf-b814-83f4e2cab7ac" Nov 03 02:45:06.264371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:06.264348 5463 scope.go:122] "RemoveContainer" containerID="2834549cc34c33417951105cb910246fa23922034708de32eb7014402f67b631" Nov 03 02:45:06.264568 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:06.264550 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc_odh-ai-gateway-infra(a70245c9-bbc6-4be9-b59a-783f65e3dae8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" podUID="a70245c9-bbc6-4be9-b59a-783f65e3dae8" Nov 03 02:45:08.865045 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:08.865006 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:12.368724 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:12.368687 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:13.008176 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:13.008136 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:45:13.667990 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:13.667952 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:14.436577 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:14.436550 5463 scope.go:122] "RemoveContainer" containerID="648d29f72c95708831e1f40bf350b3316ae903c3bad48891be5511fb7619c392" Nov 03 02:45:15.339520 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:15.339487 5463 generic.go:356] "Generic (PLEG): container finished" podID="e47bcd1a-f924-497d-8cb1-eaf0839d354f" containerID="bdd0b386d35a4e18cb843f888148dab7ebf43b515abe690d7decacf0598c2ef4" exitCode=60 Nov 03 02:45:15.339946 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:15.339555 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq" event={"ID":"e47bcd1a-f924-497d-8cb1-eaf0839d354f","Type":"ContainerDied","Data":"bdd0b386d35a4e18cb843f888148dab7ebf43b515abe690d7decacf0598c2ef4"} Nov 03 02:45:15.339946 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:15.339636 5463 scope.go:122] "RemoveContainer" containerID="648d29f72c95708831e1f40bf350b3316ae903c3bad48891be5511fb7619c392" Nov 03 02:45:15.340067 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:15.339969 5463 scope.go:122] "RemoveContainer" containerID="bdd0b386d35a4e18cb843f888148dab7ebf43b515abe690d7decacf0598c2ef4" Nov 03 02:45:15.340309 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:15.340288 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq_odh-ai-gateway-infra(e47bcd1a-f924-497d-8cb1-eaf0839d354f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq" podUID="e47bcd1a-f924-497d-8cb1-eaf0839d354f" Nov 03 02:45:15.435900 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:15.435867 5463 scope.go:122] "RemoveContainer" containerID="7cac496cc64d2167a0cfdfc0ff6af9b8301e00906653ecd3656a3b489d62b591" Nov 03 02:45:16.348745 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:16.348715 5463 generic.go:356] "Generic (PLEG): container finished" podID="d02969c8-4434-49f0-8d59-b45008195cbf" containerID="84ed70c3f13a9aac8067d8e4c790df507b28995f6ef671a82ba12349465289a8" exitCode=60 Nov 03 02:45:16.349222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:16.348793 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm" event={"ID":"d02969c8-4434-49f0-8d59-b45008195cbf","Type":"ContainerDied","Data":"84ed70c3f13a9aac8067d8e4c790df507b28995f6ef671a82ba12349465289a8"} Nov 03 02:45:16.349222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:16.348851 5463 scope.go:122] "RemoveContainer" containerID="7cac496cc64d2167a0cfdfc0ff6af9b8301e00906653ecd3656a3b489d62b591" Nov 03 02:45:16.349334 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:16.349267 5463 scope.go:122] "RemoveContainer" containerID="84ed70c3f13a9aac8067d8e4c790df507b28995f6ef671a82ba12349465289a8" Nov 03 02:45:16.349633 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:16.349608 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm_odh-ai-gateway-infra(d02969c8-4434-49f0-8d59-b45008195cbf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm" podUID="d02969c8-4434-49f0-8d59-b45008195cbf" Nov 03 02:45:16.435789 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:16.435769 5463 scope.go:122] "RemoveContainer" containerID="4ba82e231249f1c0cc31d5a2e2a6023316a3931d53fd15bc7c0d648aa0a3c4d2" Nov 03 02:45:16.465914 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:16.465663 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq"] Nov 03 02:45:16.632376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:16.632351 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq" Nov 03 02:45:16.814963 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:16.814886 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e47bcd1a-f924-497d-8cb1-eaf0839d354f-kube-api-access-fr2mz\" (UniqueName: \"kubernetes.io/projected/e47bcd1a-f924-497d-8cb1-eaf0839d354f-kube-api-access-fr2mz\") pod \"e47bcd1a-f924-497d-8cb1-eaf0839d354f\" (UID: \"e47bcd1a-f924-497d-8cb1-eaf0839d354f\") " Nov 03 02:45:16.815124 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:16.814974 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e47bcd1a-f924-497d-8cb1-eaf0839d354f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e47bcd1a-f924-497d-8cb1-eaf0839d354f-maas-api-service-ca\") pod \"e47bcd1a-f924-497d-8cb1-eaf0839d354f\" (UID: \"e47bcd1a-f924-497d-8cb1-eaf0839d354f\") " Nov 03 02:45:16.815377 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:16.815355 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e47bcd1a-f924-497d-8cb1-eaf0839d354f-maas-api-service-ca" pod "e47bcd1a-f924-497d-8cb1-eaf0839d354f" (UID: "e47bcd1a-f924-497d-8cb1-eaf0839d354f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:45:16.815535 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:16.815515 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e47bcd1a-f924-497d-8cb1-eaf0839d354f-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:16.817239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:16.817208 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47bcd1a-f924-497d-8cb1-eaf0839d354f-kube-api-access-fr2mz" pod "e47bcd1a-f924-497d-8cb1-eaf0839d354f" (UID: "e47bcd1a-f924-497d-8cb1-eaf0839d354f"). InnerVolumeSpecName "kube-api-access-fr2mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:16.916620 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:16.916593 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr2mz\" (UniqueName: \"kubernetes.io/projected/e47bcd1a-f924-497d-8cb1-eaf0839d354f-kube-api-access-fr2mz\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:17.362901 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:17.362865 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq" Nov 03 02:45:17.363381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:17.362868 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq" event={"ID":"e47bcd1a-f924-497d-8cb1-eaf0839d354f","Type":"ContainerDied","Data":"5239b51d1bd35166ad9ff7b4cdc751d14a3a927d744039440ec0e6162c601e42"} Nov 03 02:45:17.363381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:17.363033 5463 scope.go:122] "RemoveContainer" containerID="bdd0b386d35a4e18cb843f888148dab7ebf43b515abe690d7decacf0598c2ef4" Nov 03 02:45:17.365240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:17.365174 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" event={"ID":"42a5260c-8fdb-4b04-b48c-0d148fc8612b","Type":"ContainerStarted","Data":"f5ec1f785264f014980ff5a85ac810b8ca9446f9ee1bb7a01d0890ccc3d9b88b"} Nov 03 02:45:17.435977 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:17.435957 5463 scope.go:122] "RemoveContainer" containerID="53753c0747a1e4c10507e41fcd997bea655be82caca7987af4f988b7a3909f76" Nov 03 02:45:17.479463 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:17.479421 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm"] Nov 03 02:45:17.532116 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:17.532083 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq"] Nov 03 02:45:17.535217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:17.535174 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a4683-29894565-6mtdq"] Nov 03 02:45:17.679782 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:17.679759 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm" Nov 03 02:45:17.826825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:17.826801 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d02969c8-4434-49f0-8d59-b45008195cbf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02969c8-4434-49f0-8d59-b45008195cbf-maas-api-service-ca\") pod \"d02969c8-4434-49f0-8d59-b45008195cbf\" (UID: \"d02969c8-4434-49f0-8d59-b45008195cbf\") " Nov 03 02:45:17.827072 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:17.827056 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d02969c8-4434-49f0-8d59-b45008195cbf-kube-api-access-vhtcf\" (UniqueName: \"kubernetes.io/projected/d02969c8-4434-49f0-8d59-b45008195cbf-kube-api-access-vhtcf\") pod \"d02969c8-4434-49f0-8d59-b45008195cbf\" (UID: \"d02969c8-4434-49f0-8d59-b45008195cbf\") " Nov 03 02:45:17.827130 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:17.827110 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d02969c8-4434-49f0-8d59-b45008195cbf-maas-api-service-ca" pod "d02969c8-4434-49f0-8d59-b45008195cbf" (UID: "d02969c8-4434-49f0-8d59-b45008195cbf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:45:17.827412 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:17.827396 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02969c8-4434-49f0-8d59-b45008195cbf-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:17.847817 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:17.847790 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02969c8-4434-49f0-8d59-b45008195cbf-kube-api-access-vhtcf" pod "d02969c8-4434-49f0-8d59-b45008195cbf" (UID: "d02969c8-4434-49f0-8d59-b45008195cbf"). InnerVolumeSpecName "kube-api-access-vhtcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:17.928483 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:17.928412 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhtcf\" (UniqueName: \"kubernetes.io/projected/d02969c8-4434-49f0-8d59-b45008195cbf-kube-api-access-vhtcf\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:18.359015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:18.358929 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:18.373998 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:18.373969 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" event={"ID":"3d5e25d2-cf73-44b8-82f8-5789bedcc9e1","Type":"ContainerStarted","Data":"e079c4a7eabc74c24fc0ea632a5895cfb6bb450dbde42ede50e05d0f29919df0"} Nov 03 02:45:18.375463 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:18.375441 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm" Nov 03 02:45:18.375463 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:18.375450 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm" event={"ID":"d02969c8-4434-49f0-8d59-b45008195cbf","Type":"ContainerDied","Data":"cb3391cc1f91576e3ae119ba5b2f08bfbfb0604a4169b1ce18b66a0281104e09"} Nov 03 02:45:18.375635 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:18.375507 5463 scope.go:122] "RemoveContainer" containerID="84ed70c3f13a9aac8067d8e4c790df507b28995f6ef671a82ba12349465289a8" Nov 03 02:45:18.378981 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:18.378960 5463 generic.go:356] "Generic (PLEG): container finished" podID="42a5260c-8fdb-4b04-b48c-0d148fc8612b" containerID="f5ec1f785264f014980ff5a85ac810b8ca9446f9ee1bb7a01d0890ccc3d9b88b" exitCode=60 Nov 03 02:45:18.379080 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:18.379007 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" event={"ID":"42a5260c-8fdb-4b04-b48c-0d148fc8612b","Type":"ContainerDied","Data":"f5ec1f785264f014980ff5a85ac810b8ca9446f9ee1bb7a01d0890ccc3d9b88b"} Nov 03 02:45:18.379309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:18.379296 5463 scope.go:122] "RemoveContainer" containerID="f5ec1f785264f014980ff5a85ac810b8ca9446f9ee1bb7a01d0890ccc3d9b88b" Nov 03 02:45:18.379523 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:18.379507 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm_odh-ai-gateway-infra(42a5260c-8fdb-4b04-b48c-0d148fc8612b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" podUID="42a5260c-8fdb-4b04-b48c-0d148fc8612b" Nov 03 02:45:18.402109 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:18.402039 5463 scope.go:122] "RemoveContainer" containerID="4ba82e231249f1c0cc31d5a2e2a6023316a3931d53fd15bc7c0d648aa0a3c4d2" Nov 03 02:45:18.440240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:18.435594 5463 scope.go:122] "RemoveContainer" containerID="2834549cc34c33417951105cb910246fa23922034708de32eb7014402f67b631" Nov 03 02:45:18.440240 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:18.435723 5463 scope.go:122] "RemoveContainer" containerID="112f7e7505f3d6f14d8c2318ee78ba930da59f79fabb18a4bdeee2bb1783c37c" Nov 03 02:45:18.448919 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:18.448564 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e47bcd1a-f924-497d-8cb1-eaf0839d354f" path="/var/lib/kubelet/pods/e47bcd1a-f924-497d-8cb1-eaf0839d354f/volumes" Nov 03 02:45:18.654319 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:18.654289 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm"] Nov 03 02:45:18.726967 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:18.726938 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm"] Nov 03 02:45:18.730406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:18.730379 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9aebe8-29894565-whkvm"] Nov 03 02:45:19.390019 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:19.389993 5463 generic.go:356] "Generic (PLEG): container finished" podID="3d5e25d2-cf73-44b8-82f8-5789bedcc9e1" containerID="e079c4a7eabc74c24fc0ea632a5895cfb6bb450dbde42ede50e05d0f29919df0" exitCode=60 Nov 03 02:45:19.390409 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:19.390066 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" event={"ID":"3d5e25d2-cf73-44b8-82f8-5789bedcc9e1","Type":"ContainerDied","Data":"e079c4a7eabc74c24fc0ea632a5895cfb6bb450dbde42ede50e05d0f29919df0"} Nov 03 02:45:19.390409 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:19.390132 5463 scope.go:122] "RemoveContainer" containerID="53753c0747a1e4c10507e41fcd997bea655be82caca7987af4f988b7a3909f76" Nov 03 02:45:19.391609 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:19.390578 5463 scope.go:122] "RemoveContainer" containerID="e079c4a7eabc74c24fc0ea632a5895cfb6bb450dbde42ede50e05d0f29919df0" Nov 03 02:45:19.391609 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:19.390937 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8_odh-ai-gateway-infra(3d5e25d2-cf73-44b8-82f8-5789bedcc9e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" podUID="3d5e25d2-cf73-44b8-82f8-5789bedcc9e1" Nov 03 02:45:19.393069 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:19.393028 5463 generic.go:356] "Generic (PLEG): container finished" podID="da74f893-34d6-43bf-b814-83f4e2cab7ac" containerID="8d279eeb257c99624984089e3873ff0ed14da873b2653a6261b4b62d1097021d" exitCode=60 Nov 03 02:45:19.393175 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:19.393105 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" event={"ID":"da74f893-34d6-43bf-b814-83f4e2cab7ac","Type":"ContainerDied","Data":"8d279eeb257c99624984089e3873ff0ed14da873b2653a6261b4b62d1097021d"} Nov 03 02:45:19.394175 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:19.394144 5463 scope.go:122] "RemoveContainer" containerID="8d279eeb257c99624984089e3873ff0ed14da873b2653a6261b4b62d1097021d" Nov 03 02:45:19.394509 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:19.394483 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894565-c4j52_odh-ai-gateway-infra(da74f893-34d6-43bf-b814-83f4e2cab7ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" podUID="da74f893-34d6-43bf-b814-83f4e2cab7ac" Nov 03 02:45:19.397431 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:19.397408 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" event={"ID":"a70245c9-bbc6-4be9-b59a-783f65e3dae8","Type":"ContainerStarted","Data":"4d0765e91c500d89a8037382ee57694ad48d818c6e07e8818140e8c3f0da0da9"} Nov 03 02:45:19.442985 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:19.442327 5463 scope.go:122] "RemoveContainer" containerID="112f7e7505f3d6f14d8c2318ee78ba930da59f79fabb18a4bdeee2bb1783c37c" Nov 03 02:45:19.503254 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:19.502374 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8"] Nov 03 02:45:19.577750 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:19.577724 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" Nov 03 02:45:19.646590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:19.646562 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a5260c-8fdb-4b04-b48c-0d148fc8612b-kube-api-access-xgbj5\" (UniqueName: \"kubernetes.io/projected/42a5260c-8fdb-4b04-b48c-0d148fc8612b-kube-api-access-xgbj5\") pod \"42a5260c-8fdb-4b04-b48c-0d148fc8612b\" (UID: \"42a5260c-8fdb-4b04-b48c-0d148fc8612b\") " Nov 03 02:45:19.646762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:19.646739 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42a5260c-8fdb-4b04-b48c-0d148fc8612b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a5260c-8fdb-4b04-b48c-0d148fc8612b-maas-api-service-ca\") pod \"42a5260c-8fdb-4b04-b48c-0d148fc8612b\" (UID: \"42a5260c-8fdb-4b04-b48c-0d148fc8612b\") " Nov 03 02:45:19.647145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:19.647121 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a5260c-8fdb-4b04-b48c-0d148fc8612b-maas-api-service-ca" pod "42a5260c-8fdb-4b04-b48c-0d148fc8612b" (UID: "42a5260c-8fdb-4b04-b48c-0d148fc8612b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:45:19.647371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:19.647353 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a5260c-8fdb-4b04-b48c-0d148fc8612b-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:19.648811 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:19.648780 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a5260c-8fdb-4b04-b48c-0d148fc8612b-kube-api-access-xgbj5" pod "42a5260c-8fdb-4b04-b48c-0d148fc8612b" (UID: "42a5260c-8fdb-4b04-b48c-0d148fc8612b"). InnerVolumeSpecName "kube-api-access-xgbj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:19.748500 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:19.748467 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgbj5\" (UniqueName: \"kubernetes.io/projected/42a5260c-8fdb-4b04-b48c-0d148fc8612b-kube-api-access-xgbj5\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:20.410015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.409981 5463 generic.go:356] "Generic (PLEG): container finished" podID="a70245c9-bbc6-4be9-b59a-783f65e3dae8" containerID="4d0765e91c500d89a8037382ee57694ad48d818c6e07e8818140e8c3f0da0da9" exitCode=60 Nov 03 02:45:20.410517 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.410056 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" event={"ID":"a70245c9-bbc6-4be9-b59a-783f65e3dae8","Type":"ContainerDied","Data":"4d0765e91c500d89a8037382ee57694ad48d818c6e07e8818140e8c3f0da0da9"} Nov 03 02:45:20.410517 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.410124 5463 scope.go:122] "RemoveContainer" containerID="2834549cc34c33417951105cb910246fa23922034708de32eb7014402f67b631" Nov 03 02:45:20.410654 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.410534 5463 scope.go:122] "RemoveContainer" containerID="4d0765e91c500d89a8037382ee57694ad48d818c6e07e8818140e8c3f0da0da9" Nov 03 02:45:20.410907 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:20.410865 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc_odh-ai-gateway-infra(a70245c9-bbc6-4be9-b59a-783f65e3dae8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" podUID="a70245c9-bbc6-4be9-b59a-783f65e3dae8" Nov 03 02:45:20.411776 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.411757 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" Nov 03 02:45:20.411902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.411794 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm" event={"ID":"42a5260c-8fdb-4b04-b48c-0d148fc8612b","Type":"ContainerDied","Data":"ad72551ee703076b8bca34c932558ecd22d522401ccd0e7fdee5b0b70d37befa"} Nov 03 02:45:20.424282 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.424264 5463 scope.go:122] "RemoveContainer" containerID="f5ec1f785264f014980ff5a85ac810b8ca9446f9ee1bb7a01d0890ccc3d9b88b" Nov 03 02:45:20.445657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.445625 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d02969c8-4434-49f0-8d59-b45008195cbf" path="/var/lib/kubelet/pods/d02969c8-4434-49f0-8d59-b45008195cbf/volumes" Nov 03 02:45:20.585690 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.585657 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" Nov 03 02:45:20.641222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.641179 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc"] Nov 03 02:45:20.658456 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.658418 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d5e25d2-cf73-44b8-82f8-5789bedcc9e1-kube-api-access-whxqh\" (UniqueName: \"kubernetes.io/projected/3d5e25d2-cf73-44b8-82f8-5789bedcc9e1-kube-api-access-whxqh\") pod \"3d5e25d2-cf73-44b8-82f8-5789bedcc9e1\" (UID: \"3d5e25d2-cf73-44b8-82f8-5789bedcc9e1\") " Nov 03 02:45:20.658643 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.658514 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d5e25d2-cf73-44b8-82f8-5789bedcc9e1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d5e25d2-cf73-44b8-82f8-5789bedcc9e1-maas-api-service-ca\") pod \"3d5e25d2-cf73-44b8-82f8-5789bedcc9e1\" (UID: \"3d5e25d2-cf73-44b8-82f8-5789bedcc9e1\") " Nov 03 02:45:20.659082 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.659052 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d5e25d2-cf73-44b8-82f8-5789bedcc9e1-maas-api-service-ca" pod "3d5e25d2-cf73-44b8-82f8-5789bedcc9e1" (UID: "3d5e25d2-cf73-44b8-82f8-5789bedcc9e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:45:20.659347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.659326 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d5e25d2-cf73-44b8-82f8-5789bedcc9e1-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:20.661064 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.660994 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5e25d2-cf73-44b8-82f8-5789bedcc9e1-kube-api-access-whxqh" pod "3d5e25d2-cf73-44b8-82f8-5789bedcc9e1" (UID: "3d5e25d2-cf73-44b8-82f8-5789bedcc9e1"). InnerVolumeSpecName "kube-api-access-whxqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:20.751793 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.751714 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm"] Nov 03 02:45:20.756017 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.755991 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52"] Nov 03 02:45:20.759148 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.759127 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f7d9bb-29894565-l6vkm"] Nov 03 02:45:20.760514 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.760478 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whxqh\" (UniqueName: \"kubernetes.io/projected/3d5e25d2-cf73-44b8-82f8-5789bedcc9e1-kube-api-access-whxqh\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:20.796113 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.796074 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:20.939568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.939545 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" Nov 03 02:45:20.962659 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.962626 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da74f893-34d6-43bf-b814-83f4e2cab7ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da74f893-34d6-43bf-b814-83f4e2cab7ac-maas-api-service-ca\") pod \"da74f893-34d6-43bf-b814-83f4e2cab7ac\" (UID: \"da74f893-34d6-43bf-b814-83f4e2cab7ac\") " Nov 03 02:45:20.962789 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.962775 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da74f893-34d6-43bf-b814-83f4e2cab7ac-kube-api-access-hjth8\" (UniqueName: \"kubernetes.io/projected/da74f893-34d6-43bf-b814-83f4e2cab7ac-kube-api-access-hjth8\") pod \"da74f893-34d6-43bf-b814-83f4e2cab7ac\" (UID: \"da74f893-34d6-43bf-b814-83f4e2cab7ac\") " Nov 03 02:45:20.962993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.962969 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da74f893-34d6-43bf-b814-83f4e2cab7ac-maas-api-service-ca" pod "da74f893-34d6-43bf-b814-83f4e2cab7ac" (UID: "da74f893-34d6-43bf-b814-83f4e2cab7ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:45:20.963269 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.963254 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da74f893-34d6-43bf-b814-83f4e2cab7ac-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:20.964762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:20.964740 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da74f893-34d6-43bf-b814-83f4e2cab7ac-kube-api-access-hjth8" pod "da74f893-34d6-43bf-b814-83f4e2cab7ac" (UID: "da74f893-34d6-43bf-b814-83f4e2cab7ac"). InnerVolumeSpecName "kube-api-access-hjth8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:21.064714 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:21.064684 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjth8\" (UniqueName: \"kubernetes.io/projected/da74f893-34d6-43bf-b814-83f4e2cab7ac-kube-api-access-hjth8\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:21.421808 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:21.421777 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" Nov 03 02:45:21.422320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:21.421811 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8" event={"ID":"3d5e25d2-cf73-44b8-82f8-5789bedcc9e1","Type":"ContainerDied","Data":"91ab49d88b4e28ec9c125ac6fa8f5f024b1caa0dca08b3184881322eaf7b9e10"} Nov 03 02:45:21.422320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:21.421896 5463 scope.go:122] "RemoveContainer" containerID="e079c4a7eabc74c24fc0ea632a5895cfb6bb450dbde42ede50e05d0f29919df0" Nov 03 02:45:21.423631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:21.423603 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" Nov 03 02:45:21.423631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:21.423620 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52" event={"ID":"da74f893-34d6-43bf-b814-83f4e2cab7ac","Type":"ContainerDied","Data":"23f95ab2304c1ccfc5f543498769f16e5b2e57addef039028970cce3eaf24d43"} Nov 03 02:45:21.433832 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:21.433812 5463 scope.go:122] "RemoveContainer" containerID="8d279eeb257c99624984089e3873ff0ed14da873b2653a6261b4b62d1097021d" Nov 03 02:45:21.576568 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:21.576546 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" Nov 03 02:45:21.673068 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:21.672989 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a70245c9-bbc6-4be9-b59a-783f65e3dae8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70245c9-bbc6-4be9-b59a-783f65e3dae8-maas-api-service-ca\") pod \"a70245c9-bbc6-4be9-b59a-783f65e3dae8\" (UID: \"a70245c9-bbc6-4be9-b59a-783f65e3dae8\") " Nov 03 02:45:21.673234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:21.673168 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a70245c9-bbc6-4be9-b59a-783f65e3dae8-kube-api-access-8958w\" (UniqueName: \"kubernetes.io/projected/a70245c9-bbc6-4be9-b59a-783f65e3dae8-kube-api-access-8958w\") pod \"a70245c9-bbc6-4be9-b59a-783f65e3dae8\" (UID: \"a70245c9-bbc6-4be9-b59a-783f65e3dae8\") " Nov 03 02:45:21.673446 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:21.673421 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70245c9-bbc6-4be9-b59a-783f65e3dae8-maas-api-service-ca" pod "a70245c9-bbc6-4be9-b59a-783f65e3dae8" (UID: "a70245c9-bbc6-4be9-b59a-783f65e3dae8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:45:21.673664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:21.673633 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70245c9-bbc6-4be9-b59a-783f65e3dae8-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:21.675549 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:21.675523 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70245c9-bbc6-4be9-b59a-783f65e3dae8-kube-api-access-8958w" pod "a70245c9-bbc6-4be9-b59a-783f65e3dae8" (UID: "a70245c9-bbc6-4be9-b59a-783f65e3dae8"). InnerVolumeSpecName "kube-api-access-8958w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:21.714823 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:21.714796 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52"] Nov 03 02:45:21.717674 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:21.717649 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894565-c4j52"] Nov 03 02:45:21.774673 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:21.774628 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8958w\" (UniqueName: \"kubernetes.io/projected/a70245c9-bbc6-4be9-b59a-783f65e3dae8-kube-api-access-8958w\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:21.828801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:21.828772 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8"] Nov 03 02:45:21.832239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:21.832214 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-78b5c5-29894565-nlnd8"] Nov 03 02:45:22.437361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:22.437338 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" Nov 03 02:45:22.443599 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:22.443570 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d5e25d2-cf73-44b8-82f8-5789bedcc9e1" path="/var/lib/kubelet/pods/3d5e25d2-cf73-44b8-82f8-5789bedcc9e1/volumes" Nov 03 02:45:22.444046 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:22.444032 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42a5260c-8fdb-4b04-b48c-0d148fc8612b" path="/var/lib/kubelet/pods/42a5260c-8fdb-4b04-b48c-0d148fc8612b/volumes" Nov 03 02:45:22.444534 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:22.444509 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da74f893-34d6-43bf-b814-83f4e2cab7ac" path="/var/lib/kubelet/pods/da74f893-34d6-43bf-b814-83f4e2cab7ac/volumes" Nov 03 02:45:22.445442 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:22.445424 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc" event={"ID":"a70245c9-bbc6-4be9-b59a-783f65e3dae8","Type":"ContainerDied","Data":"510d9cf97866b420d2c5e9f560570cc9dd00d50bda7094fb654395eec9da816c"} Nov 03 02:45:22.445538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:22.445469 5463 scope.go:122] "RemoveContainer" containerID="4d0765e91c500d89a8037382ee57694ad48d818c6e07e8818140e8c3f0da0da9" Nov 03 02:45:22.604042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:22.604008 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc"] Nov 03 02:45:22.655856 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:22.655824 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-965a07-29894565-qnlwc"] Nov 03 02:45:23.194981 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:23.194946 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:24.443860 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:24.443829 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a70245c9-bbc6-4be9-b59a-783f65e3dae8" path="/var/lib/kubelet/pods/a70245c9-bbc6-4be9-b59a-783f65e3dae8/volumes" Nov 03 02:45:27.879224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:27.879162 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:28.667520 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:28.667472 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:29.256640 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:29.256611 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:29.867048 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:29.866228 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:30.042200 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:30.042156 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:32.625534 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:32.625491 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:34.158363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:34.158324 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:34.376776 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:34.376719 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:34.547235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:34.547068 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:37.354215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.350958 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:37.498322 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.498288 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:37.830535 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.830501 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:37.854406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.854372 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc"] Nov 03 02:45:37.855361 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855341 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a70245c9-bbc6-4be9-b59a-783f65e3dae8" containerName="cleanup" Nov 03 02:45:37.855440 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855364 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70245c9-bbc6-4be9-b59a-783f65e3dae8" containerName="cleanup" Nov 03 02:45:37.855440 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855398 5463 cpu_manager.go:414] "containerMap: removing container" podUID="3d5e25d2-cf73-44b8-82f8-5789bedcc9e1" containerName="cleanup" Nov 03 02:45:37.855440 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855406 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5e25d2-cf73-44b8-82f8-5789bedcc9e1" containerName="cleanup" Nov 03 02:45:37.855440 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855423 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d02969c8-4434-49f0-8d59-b45008195cbf" containerName="cleanup" Nov 03 02:45:37.855440 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855431 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d02969c8-4434-49f0-8d59-b45008195cbf" containerName="cleanup" Nov 03 02:45:37.855633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855463 5463 cpu_manager.go:414] "containerMap: removing container" podUID="da74f893-34d6-43bf-b814-83f4e2cab7ac" containerName="cleanup" Nov 03 02:45:37.855633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855471 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da74f893-34d6-43bf-b814-83f4e2cab7ac" containerName="cleanup" Nov 03 02:45:37.855633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855479 5463 cpu_manager.go:414] "containerMap: removing container" podUID="3d5e25d2-cf73-44b8-82f8-5789bedcc9e1" containerName="cleanup" Nov 03 02:45:37.855633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855487 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5e25d2-cf73-44b8-82f8-5789bedcc9e1" containerName="cleanup" Nov 03 02:45:37.855633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855498 5463 cpu_manager.go:414] "containerMap: removing container" podUID="42a5260c-8fdb-4b04-b48c-0d148fc8612b" containerName="cleanup" Nov 03 02:45:37.855633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855505 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a5260c-8fdb-4b04-b48c-0d148fc8612b" containerName="cleanup" Nov 03 02:45:37.855633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855519 5463 cpu_manager.go:414] "containerMap: removing container" podUID="42a5260c-8fdb-4b04-b48c-0d148fc8612b" containerName="cleanup" Nov 03 02:45:37.855633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855526 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a5260c-8fdb-4b04-b48c-0d148fc8612b" containerName="cleanup" Nov 03 02:45:37.855633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855537 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d02969c8-4434-49f0-8d59-b45008195cbf" containerName="cleanup" Nov 03 02:45:37.855633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855544 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d02969c8-4434-49f0-8d59-b45008195cbf" containerName="cleanup" Nov 03 02:45:37.855633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855559 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a70245c9-bbc6-4be9-b59a-783f65e3dae8" containerName="cleanup" Nov 03 02:45:37.855633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855565 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70245c9-bbc6-4be9-b59a-783f65e3dae8" containerName="cleanup" Nov 03 02:45:37.855633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855575 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d02969c8-4434-49f0-8d59-b45008195cbf" containerName="cleanup" Nov 03 02:45:37.855633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855583 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d02969c8-4434-49f0-8d59-b45008195cbf" containerName="cleanup" Nov 03 02:45:37.855633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855593 5463 cpu_manager.go:414] "containerMap: removing container" podUID="3d5e25d2-cf73-44b8-82f8-5789bedcc9e1" containerName="cleanup" Nov 03 02:45:37.855633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855600 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5e25d2-cf73-44b8-82f8-5789bedcc9e1" containerName="cleanup" Nov 03 02:45:37.855633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855615 5463 cpu_manager.go:414] "containerMap: removing container" podUID="e47bcd1a-f924-497d-8cb1-eaf0839d354f" containerName="cleanup" Nov 03 02:45:37.855633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855623 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e47bcd1a-f924-497d-8cb1-eaf0839d354f" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855644 5463 cpu_manager.go:414] "containerMap: removing container" podUID="e47bcd1a-f924-497d-8cb1-eaf0839d354f" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855651 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e47bcd1a-f924-497d-8cb1-eaf0839d354f" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855660 5463 cpu_manager.go:414] "containerMap: removing container" podUID="da74f893-34d6-43bf-b814-83f4e2cab7ac" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855666 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da74f893-34d6-43bf-b814-83f4e2cab7ac" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855681 5463 cpu_manager.go:414] "containerMap: removing container" podUID="42a5260c-8fdb-4b04-b48c-0d148fc8612b" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855688 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a5260c-8fdb-4b04-b48c-0d148fc8612b" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855827 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a5260c-8fdb-4b04-b48c-0d148fc8612b" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855838 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d02969c8-4434-49f0-8d59-b45008195cbf" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855848 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70245c9-bbc6-4be9-b59a-783f65e3dae8" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855857 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d02969c8-4434-49f0-8d59-b45008195cbf" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855864 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70245c9-bbc6-4be9-b59a-783f65e3dae8" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855871 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5e25d2-cf73-44b8-82f8-5789bedcc9e1" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855878 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e47bcd1a-f924-497d-8cb1-eaf0839d354f" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855888 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="da74f893-34d6-43bf-b814-83f4e2cab7ac" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855897 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a5260c-8fdb-4b04-b48c-0d148fc8612b" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855906 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e47bcd1a-f924-497d-8cb1-eaf0839d354f" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855917 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="da74f893-34d6-43bf-b814-83f4e2cab7ac" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855932 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5e25d2-cf73-44b8-82f8-5789bedcc9e1" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855942 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d02969c8-4434-49f0-8d59-b45008195cbf" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855955 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="da74f893-34d6-43bf-b814-83f4e2cab7ac" containerName="cleanup" Nov 03 02:45:37.856299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.855964 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5e25d2-cf73-44b8-82f8-5789bedcc9e1" containerName="cleanup" Nov 03 02:45:37.861778 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.861744 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc" Nov 03 02:45:37.865824 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.865798 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec0819e-4232-43a9-820e-3a7b65b7ac9d-plugins-config-volume\") pod \"payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc\" (UID: \"5ec0819e-4232-43a9-820e-3a7b65b7ac9d\") " pod="openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc" Nov 03 02:45:37.866122 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.866098 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6rfp\" (UniqueName: \"kubernetes.io/projected/5ec0819e-4232-43a9-820e-3a7b65b7ac9d-kube-api-access-p6rfp\") pod \"payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc\" (UID: \"5ec0819e-4232-43a9-820e-3a7b65b7ac9d\") " pod="openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc" Nov 03 02:45:37.913431 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.913399 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc"] Nov 03 02:45:37.959316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.957074 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-c8ee6b\"" Nov 03 02:45:37.959316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.957247 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-c8ee6b-dockercfg-mnqk6\"" Nov 03 02:45:37.967486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.967452 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec0819e-4232-43a9-820e-3a7b65b7ac9d-plugins-config-volume\") pod \"payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc\" (UID: \"5ec0819e-4232-43a9-820e-3a7b65b7ac9d\") " pod="openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc" Nov 03 02:45:37.967706 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.967689 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6rfp\" (UniqueName: \"kubernetes.io/projected/5ec0819e-4232-43a9-820e-3a7b65b7ac9d-kube-api-access-p6rfp\") pod \"payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc\" (UID: \"5ec0819e-4232-43a9-820e-3a7b65b7ac9d\") " pod="openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc" Nov 03 02:45:37.969271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:37.969247 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec0819e-4232-43a9-820e-3a7b65b7ac9d-plugins-config-volume\") pod \"payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc\" (UID: \"5ec0819e-4232-43a9-820e-3a7b65b7ac9d\") " pod="openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc" Nov 03 02:45:38.025969 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.025926 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6rfp\" (UniqueName: \"kubernetes.io/projected/5ec0819e-4232-43a9-820e-3a7b65b7ac9d-kube-api-access-p6rfp\") pod \"payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc\" (UID: \"5ec0819e-4232-43a9-820e-3a7b65b7ac9d\") " pod="openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc" Nov 03 02:45:38.047651 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.047617 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g"] Nov 03 02:45:38.048840 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.048814 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a70245c9-bbc6-4be9-b59a-783f65e3dae8" containerName="cleanup" Nov 03 02:45:38.048840 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.048842 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70245c9-bbc6-4be9-b59a-783f65e3dae8" containerName="cleanup" Nov 03 02:45:38.049006 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.048866 5463 cpu_manager.go:414] "containerMap: removing container" podUID="e47bcd1a-f924-497d-8cb1-eaf0839d354f" containerName="cleanup" Nov 03 02:45:38.049006 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.048876 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e47bcd1a-f924-497d-8cb1-eaf0839d354f" containerName="cleanup" Nov 03 02:45:38.049006 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.048900 5463 cpu_manager.go:414] "containerMap: removing container" podUID="da74f893-34d6-43bf-b814-83f4e2cab7ac" containerName="cleanup" Nov 03 02:45:38.049006 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.048908 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da74f893-34d6-43bf-b814-83f4e2cab7ac" containerName="cleanup" Nov 03 02:45:38.049186 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.049057 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70245c9-bbc6-4be9-b59a-783f65e3dae8" containerName="cleanup" Nov 03 02:45:38.049186 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.049070 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a5260c-8fdb-4b04-b48c-0d148fc8612b" containerName="cleanup" Nov 03 02:45:38.049186 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.049076 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e47bcd1a-f924-497d-8cb1-eaf0839d354f" containerName="cleanup" Nov 03 02:45:38.052818 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.052774 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g" Nov 03 02:45:38.068842 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.068790 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lplm6\" (UniqueName: \"kubernetes.io/projected/64bcbb35-8a22-43a4-beae-fbd0420e5bdd-kube-api-access-lplm6\") pod \"payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g\" (UID: \"64bcbb35-8a22-43a4-beae-fbd0420e5bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g" Nov 03 02:45:38.069009 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.068949 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64bcbb35-8a22-43a4-beae-fbd0420e5bdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g\" (UID: \"64bcbb35-8a22-43a4-beae-fbd0420e5bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g" Nov 03 02:45:38.105241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.105134 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g"] Nov 03 02:45:38.170488 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.170452 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lplm6\" (UniqueName: \"kubernetes.io/projected/64bcbb35-8a22-43a4-beae-fbd0420e5bdd-kube-api-access-lplm6\") pod \"payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g\" (UID: \"64bcbb35-8a22-43a4-beae-fbd0420e5bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g" Nov 03 02:45:38.170633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.170556 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64bcbb35-8a22-43a4-beae-fbd0420e5bdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g\" (UID: \"64bcbb35-8a22-43a4-beae-fbd0420e5bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g" Nov 03 02:45:38.171306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.171287 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64bcbb35-8a22-43a4-beae-fbd0420e5bdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g\" (UID: \"64bcbb35-8a22-43a4-beae-fbd0420e5bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g" Nov 03 02:45:38.228348 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.228323 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lplm6\" (UniqueName: \"kubernetes.io/projected/64bcbb35-8a22-43a4-beae-fbd0420e5bdd-kube-api-access-lplm6\") pod \"payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g\" (UID: \"64bcbb35-8a22-43a4-beae-fbd0420e5bdd\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g" Nov 03 02:45:38.276518 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.276489 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc" Nov 03 02:45:38.380318 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.380239 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g" Nov 03 02:45:38.651444 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:45:38.651413 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec0819e_4232_43a9_820e_3a7b65b7ac9d.slice/crio-e35aa638fc1ca9bcb67b1b0152e83afa99452ce73380b7b86d855578fc06fbb9 WatchSource:0}: Error finding container e35aa638fc1ca9bcb67b1b0152e83afa99452ce73380b7b86d855578fc06fbb9: Status 404 returned error can't find the container with id e35aa638fc1ca9bcb67b1b0152e83afa99452ce73380b7b86d855578fc06fbb9 Nov 03 02:45:38.654300 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.654276 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc"] Nov 03 02:45:38.765009 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:45:38.764982 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64bcbb35_8a22_43a4_beae_fbd0420e5bdd.slice/crio-36ccfb53fd5b5e004c01be16ef80c01818c2b8036f7a8691aacf06cba42420c7 WatchSource:0}: Error finding container 36ccfb53fd5b5e004c01be16ef80c01818c2b8036f7a8691aacf06cba42420c7: Status 404 returned error can't find the container with id 36ccfb53fd5b5e004c01be16ef80c01818c2b8036f7a8691aacf06cba42420c7 Nov 03 02:45:38.767752 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.767732 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g"] Nov 03 02:45:38.941910 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:38.941810 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k"] Nov 03 02:45:39.007719 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:39.007670 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:39.582829 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:39.582795 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc" event={"ID":"5ec0819e-4232-43a9-820e-3a7b65b7ac9d","Type":"ContainerStarted","Data":"a705845d5a90b977261816a518e7d615358e266c9acc967852e39dfc5c363a5c"} Nov 03 02:45:39.583323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:39.582883 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc" event={"ID":"5ec0819e-4232-43a9-820e-3a7b65b7ac9d","Type":"ContainerStarted","Data":"e35aa638fc1ca9bcb67b1b0152e83afa99452ce73380b7b86d855578fc06fbb9"} Nov 03 02:45:39.583323 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:39.582919 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc" Nov 03 02:45:39.584918 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:39.584883 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g" event={"ID":"64bcbb35-8a22-43a4-beae-fbd0420e5bdd","Type":"ContainerStarted","Data":"762fa72ffa6f5caa29664846acee6f3be0ff27961b236490f7ec660476c2b7f6"} Nov 03 02:45:39.585059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:39.584946 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g" event={"ID":"64bcbb35-8a22-43a4-beae-fbd0420e5bdd","Type":"ContainerStarted","Data":"36ccfb53fd5b5e004c01be16ef80c01818c2b8036f7a8691aacf06cba42420c7"} Nov 03 02:45:39.585059 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:39.584968 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g" Nov 03 02:45:39.698276 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:39.698223 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc" podStartSLOduration=2.698209061 podStartE2EDuration="2.698209061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:39.695557592 +0000 UTC m=+3221.379666748" watchObservedRunningTime="2026-11-03 02:45:39.698209061 +0000 UTC m=+3221.382318222" Nov 03 02:45:39.813142 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:39.813084 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g" podStartSLOduration=1.813068648 podStartE2EDuration="1.813068648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:39.80988877 +0000 UTC m=+3221.493997928" watchObservedRunningTime="2026-11-03 02:45:39.813068648 +0000 UTC m=+3221.497177802" Nov 03 02:45:42.720699 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:42.720667 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:43.009299 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:43.009219 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:45:45.652063 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:45.652025 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:47.509867 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:47.509833 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:50.597630 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:50.597599 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc" Nov 03 02:45:50.598105 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:50.597668 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g" Nov 03 02:45:52.169808 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:52.169775 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:53.971457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:53.971417 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" podUID="5effe46b-a96c-46ab-ae45-188c31537fbd" containerName="main" containerID="cri-o://b03ba6a91609d9a0cafc23dcff960ee1ceac37e14db76888a5cc228103496409" gracePeriod=45 Nov 03 02:45:54.246160 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.246136 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:45:54.376602 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.376569 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5effe46b-a96c-46ab-ae45-188c31537fbd-tls-certs\" (UniqueName: \"kubernetes.io/secret/5effe46b-a96c-46ab-ae45-188c31537fbd-tls-certs\") pod \"5effe46b-a96c-46ab-ae45-188c31537fbd\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " Nov 03 02:45:54.376803 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.376785 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-tmp-dir\") pod \"5effe46b-a96c-46ab-ae45-188c31537fbd\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " Nov 03 02:45:54.376869 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.376846 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-dshm\") pod \"5effe46b-a96c-46ab-ae45-188c31537fbd\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " Nov 03 02:45:54.376977 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.376960 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-model-cache\") pod \"5effe46b-a96c-46ab-ae45-188c31537fbd\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " Nov 03 02:45:54.377090 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.377075 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-home\" (UniqueName: \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-home\") pod \"5effe46b-a96c-46ab-ae45-188c31537fbd\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " Nov 03 02:45:54.377090 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.377076 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-tmp-dir" pod "5effe46b-a96c-46ab-ae45-188c31537fbd" (UID: "5effe46b-a96c-46ab-ae45-188c31537fbd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:45:54.377242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.377222 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5effe46b-a96c-46ab-ae45-188c31537fbd-kube-api-access-6q8bl\" (UniqueName: \"kubernetes.io/projected/5effe46b-a96c-46ab-ae45-188c31537fbd-kube-api-access-6q8bl\") pod \"5effe46b-a96c-46ab-ae45-188c31537fbd\" (UID: \"5effe46b-a96c-46ab-ae45-188c31537fbd\") " Nov 03 02:45:54.377321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.377252 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-model-cache" pod "5effe46b-a96c-46ab-ae45-188c31537fbd" (UID: "5effe46b-a96c-46ab-ae45-188c31537fbd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:45:54.377384 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.377349 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-home" pod "5effe46b-a96c-46ab-ae45-188c31537fbd" (UID: "5effe46b-a96c-46ab-ae45-188c31537fbd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:45:54.377804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.377785 5463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-model-cache\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:54.377910 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.377807 5463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-home\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:54.377910 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.377822 5463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-tmp-dir\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:54.379720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.379700 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-dshm" pod "5effe46b-a96c-46ab-ae45-188c31537fbd" (UID: "5effe46b-a96c-46ab-ae45-188c31537fbd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:45:54.379826 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.379756 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5effe46b-a96c-46ab-ae45-188c31537fbd-tls-certs" pod "5effe46b-a96c-46ab-ae45-188c31537fbd" (UID: "5effe46b-a96c-46ab-ae45-188c31537fbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:45:54.379890 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.379816 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5effe46b-a96c-46ab-ae45-188c31537fbd-kube-api-access-6q8bl" pod "5effe46b-a96c-46ab-ae45-188c31537fbd" (UID: "5effe46b-a96c-46ab-ae45-188c31537fbd"). InnerVolumeSpecName "kube-api-access-6q8bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:54.478806 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.478737 5463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5effe46b-a96c-46ab-ae45-188c31537fbd-dshm\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:54.478806 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.478766 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q8bl\" (UniqueName: \"kubernetes.io/projected/5effe46b-a96c-46ab-ae45-188c31537fbd-kube-api-access-6q8bl\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:54.478806 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.478781 5463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5effe46b-a96c-46ab-ae45-188c31537fbd-tls-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:54.694755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.694718 5463 generic.go:356] "Generic (PLEG): container finished" podID="5effe46b-a96c-46ab-ae45-188c31537fbd" containerID="b03ba6a91609d9a0cafc23dcff960ee1ceac37e14db76888a5cc228103496409" exitCode=0 Nov 03 02:45:54.694951 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.694776 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" Nov 03 02:45:54.694951 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.694808 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" event={"ID":"5effe46b-a96c-46ab-ae45-188c31537fbd","Type":"ContainerDied","Data":"b03ba6a91609d9a0cafc23dcff960ee1ceac37e14db76888a5cc228103496409"} Nov 03 02:45:54.694951 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.694863 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k" event={"ID":"5effe46b-a96c-46ab-ae45-188c31537fbd","Type":"ContainerDied","Data":"9c8ce3f44f2220a452e1310853918a11e6aef8cc0629085a46d6b5e5f04a8e41"} Nov 03 02:45:54.694951 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.694876 5463 scope.go:122] "RemoveContainer" containerID="b03ba6a91609d9a0cafc23dcff960ee1ceac37e14db76888a5cc228103496409" Nov 03 02:45:54.704955 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.704937 5463 scope.go:122] "RemoveContainer" containerID="b03ba6a91609d9a0cafc23dcff960ee1ceac37e14db76888a5cc228103496409" Nov 03 02:45:54.705246 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:45:54.705220 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03ba6a91609d9a0cafc23dcff960ee1ceac37e14db76888a5cc228103496409\": container with ID starting with b03ba6a91609d9a0cafc23dcff960ee1ceac37e14db76888a5cc228103496409 not found: ID does not exist" containerID="b03ba6a91609d9a0cafc23dcff960ee1ceac37e14db76888a5cc228103496409" Nov 03 02:45:54.705309 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.705255 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03ba6a91609d9a0cafc23dcff960ee1ceac37e14db76888a5cc228103496409"} err="failed to get container status \"b03ba6a91609d9a0cafc23dcff960ee1ceac37e14db76888a5cc228103496409\": rpc error: code = NotFound desc = could not find container \"b03ba6a91609d9a0cafc23dcff960ee1ceac37e14db76888a5cc228103496409\": container with ID starting with b03ba6a91609d9a0cafc23dcff960ee1ceac37e14db76888a5cc228103496409 not found: ID does not exist" Nov 03 02:45:54.861636 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.861542 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k"] Nov 03 02:45:54.864942 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:54.864917 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3a4683/auth-test-model-3a4683-kserve-84bc685497-rgm4k"] Nov 03 02:45:56.443533 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.443503 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5effe46b-a96c-46ab-ae45-188c31537fbd" path="/var/lib/kubelet/pods/5effe46b-a96c-46ab-ae45-188c31537fbd/volumes" Nov 03 02:45:56.675171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.675139 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l"] Nov 03 02:45:56.675812 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.675796 5463 cpu_manager.go:414] "containerMap: removing container" podUID="5effe46b-a96c-46ab-ae45-188c31537fbd" containerName="main" Nov 03 02:45:56.675858 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.675813 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5effe46b-a96c-46ab-ae45-188c31537fbd" containerName="main" Nov 03 02:45:56.675947 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.675938 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="5effe46b-a96c-46ab-ae45-188c31537fbd" containerName="main" Nov 03 02:45:56.680973 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.680954 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" Nov 03 02:45:56.731649 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.731572 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l"] Nov 03 02:45:56.733502 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.733479 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-038c21-serving-cert\"" Nov 03 02:45:56.811644 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.811609 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66888dec-a1ab-4541-9c61-58014ece2263-metrics-certs\") pod \"maas-api-e2e-explicit-038c21-6479f66779-drf7l\" (UID: \"66888dec-a1ab-4541-9c61-58014ece2263\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" Nov 03 02:45:56.811838 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.811793 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66888dec-a1ab-4541-9c61-58014ece2263-openshift-service-ca\") pod \"maas-api-e2e-explicit-038c21-6479f66779-drf7l\" (UID: \"66888dec-a1ab-4541-9c61-58014ece2263\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" Nov 03 02:45:56.811915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.811879 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66888dec-a1ab-4541-9c61-58014ece2263-maas-api-tls\") pod \"maas-api-e2e-explicit-038c21-6479f66779-drf7l\" (UID: \"66888dec-a1ab-4541-9c61-58014ece2263\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" Nov 03 02:45:56.812026 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.812005 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nk64\" (UniqueName: \"kubernetes.io/projected/66888dec-a1ab-4541-9c61-58014ece2263-kube-api-access-2nk64\") pod \"maas-api-e2e-explicit-038c21-6479f66779-drf7l\" (UID: \"66888dec-a1ab-4541-9c61-58014ece2263\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" Nov 03 02:45:56.882618 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.882576 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:56.913476 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.913443 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66888dec-a1ab-4541-9c61-58014ece2263-maas-api-tls\") pod \"maas-api-e2e-explicit-038c21-6479f66779-drf7l\" (UID: \"66888dec-a1ab-4541-9c61-58014ece2263\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" Nov 03 02:45:56.913603 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.913577 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nk64\" (UniqueName: \"kubernetes.io/projected/66888dec-a1ab-4541-9c61-58014ece2263-kube-api-access-2nk64\") pod \"maas-api-e2e-explicit-038c21-6479f66779-drf7l\" (UID: \"66888dec-a1ab-4541-9c61-58014ece2263\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" Nov 03 02:45:56.913650 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.913625 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66888dec-a1ab-4541-9c61-58014ece2263-metrics-certs\") pod \"maas-api-e2e-explicit-038c21-6479f66779-drf7l\" (UID: \"66888dec-a1ab-4541-9c61-58014ece2263\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" Nov 03 02:45:56.913764 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.913747 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66888dec-a1ab-4541-9c61-58014ece2263-openshift-service-ca\") pod \"maas-api-e2e-explicit-038c21-6479f66779-drf7l\" (UID: \"66888dec-a1ab-4541-9c61-58014ece2263\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" Nov 03 02:45:56.914434 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.914410 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66888dec-a1ab-4541-9c61-58014ece2263-openshift-service-ca\") pod \"maas-api-e2e-explicit-038c21-6479f66779-drf7l\" (UID: \"66888dec-a1ab-4541-9c61-58014ece2263\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" Nov 03 02:45:56.915914 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.915895 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66888dec-a1ab-4541-9c61-58014ece2263-maas-api-tls\") pod \"maas-api-e2e-explicit-038c21-6479f66779-drf7l\" (UID: \"66888dec-a1ab-4541-9c61-58014ece2263\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" Nov 03 02:45:56.916007 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.915971 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66888dec-a1ab-4541-9c61-58014ece2263-metrics-certs\") pod \"maas-api-e2e-explicit-038c21-6479f66779-drf7l\" (UID: \"66888dec-a1ab-4541-9c61-58014ece2263\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" Nov 03 02:45:56.971379 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.971348 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nk64\" (UniqueName: \"kubernetes.io/projected/66888dec-a1ab-4541-9c61-58014ece2263-kube-api-access-2nk64\") pod \"maas-api-e2e-explicit-038c21-6479f66779-drf7l\" (UID: \"66888dec-a1ab-4541-9c61-58014ece2263\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" Nov 03 02:45:56.995110 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:56.995041 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" Nov 03 02:45:57.042799 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:57.042384 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:57.192281 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:57.192244 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:57.377256 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:45:57.377226 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66888dec_a1ab_4541_9c61_58014ece2263.slice/crio-d2ce19631b77071fbd068e67d52e849b29fecb0beeb86f178488418517fa959c WatchSource:0}: Error finding container d2ce19631b77071fbd068e67d52e849b29fecb0beeb86f178488418517fa959c: Status 404 returned error can't find the container with id d2ce19631b77071fbd068e67d52e849b29fecb0beeb86f178488418517fa959c Nov 03 02:45:57.378946 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:57.378923 5463 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 02:45:57.381279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:57.381257 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l"] Nov 03 02:45:57.621991 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:57.621960 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:57.717890 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:57.717860 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" event={"ID":"66888dec-a1ab-4541-9c61-58014ece2263","Type":"ContainerStarted","Data":"d2ce19631b77071fbd068e67d52e849b29fecb0beeb86f178488418517fa959c"} Nov 03 02:45:58.726036 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:58.726000 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" event={"ID":"66888dec-a1ab-4541-9c61-58014ece2263","Type":"ContainerStarted","Data":"af9608a7a1ad615aad185814a3371019813ef0d5dc134ab3b7f250e178b25d10"} Nov 03 02:45:58.726529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:58.726215 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" Nov 03 02:45:58.843565 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:45:58.843514 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" podStartSLOduration=1.956297146 podStartE2EDuration="2.843500336s" totalImagesPullingTime="887.20319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:58.841910998 +0000 UTC m=+3240.526020153" watchObservedRunningTime="2026-11-03 02:45:58.843500336 +0000 UTC m=+3240.527609492" Nov 03 02:46:01.747363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:01.747331 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:01.913534 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:01.913481 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:04.740090 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:04.740064 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" Nov 03 02:46:06.540459 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:06.540425 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:10.292465 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:10.292361 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:11.446151 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:11.446111 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:11.891573 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:11.891463 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:12.758399 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:12.758344 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:13.077304 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:46:13.076663 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:46:13.077304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:13.076898 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:14.534283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:14.534241 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:14.873239 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:14.873142 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:15.048288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:15.048249 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:16.130661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:16.130601 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:17.645408 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:17.645369 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:17.860530 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:17.860488 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:18.022565 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:18.022483 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:18.736026 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:18.735983 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:19.356797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:19.356757 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:19.560562 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:19.560530 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:20.028936 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:20.028898 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:20.964719 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:20.964673 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:21.153332 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:21.153291 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:21.378417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:21.378333 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:21.545777 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:21.545735 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:23.755007 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:23.754972 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:23.971330 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:23.971291 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:24.148819 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:24.148787 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:26.753237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:26.753163 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:27.022967 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:27.022896 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:27.187866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:27.187829 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:31.998042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:31.998005 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:33.645135 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:33.645097 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:37.260345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:37.260297 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:40.676656 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:40.676612 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:42.547169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:42.547129 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:43.009333 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:46:43.009299 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:46:44.941563 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:44.941525 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:45.815688 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:45.815654 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:46.138924 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:46.138850 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:46.958976 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:46.958915 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994"] Nov 03 02:46:46.964097 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:46.964072 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" Nov 03 02:46:47.017152 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:47.017132 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-c8ee6b-serving-cert\"" Nov 03 02:46:47.093439 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:47.093401 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2lks\" (UniqueName: \"kubernetes.io/projected/744caa65-978b-4a72-af47-fc8db4ad7f02-kube-api-access-w2lks\") pod \"maas-api-e2e-autores-c8ee6b-6b8b548978-qt994\" (UID: \"744caa65-978b-4a72-af47-fc8db4ad7f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" Nov 03 02:46:47.094342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:47.094307 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/744caa65-978b-4a72-af47-fc8db4ad7f02-maas-api-tls\") pod \"maas-api-e2e-autores-c8ee6b-6b8b548978-qt994\" (UID: \"744caa65-978b-4a72-af47-fc8db4ad7f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" Nov 03 02:46:47.094626 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:47.094608 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/744caa65-978b-4a72-af47-fc8db4ad7f02-openshift-service-ca\") pod \"maas-api-e2e-autores-c8ee6b-6b8b548978-qt994\" (UID: \"744caa65-978b-4a72-af47-fc8db4ad7f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" Nov 03 02:46:47.094786 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:47.094739 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/744caa65-978b-4a72-af47-fc8db4ad7f02-metrics-certs\") pod \"maas-api-e2e-autores-c8ee6b-6b8b548978-qt994\" (UID: \"744caa65-978b-4a72-af47-fc8db4ad7f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" Nov 03 02:46:47.099523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:47.099503 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994"] Nov 03 02:46:47.171000 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:47.170959 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:47.195941 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:47.195912 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/744caa65-978b-4a72-af47-fc8db4ad7f02-openshift-service-ca\") pod \"maas-api-e2e-autores-c8ee6b-6b8b548978-qt994\" (UID: \"744caa65-978b-4a72-af47-fc8db4ad7f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" Nov 03 02:46:47.196090 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:47.195954 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/744caa65-978b-4a72-af47-fc8db4ad7f02-metrics-certs\") pod \"maas-api-e2e-autores-c8ee6b-6b8b548978-qt994\" (UID: \"744caa65-978b-4a72-af47-fc8db4ad7f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" Nov 03 02:46:47.196090 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:47.196036 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2lks\" (UniqueName: \"kubernetes.io/projected/744caa65-978b-4a72-af47-fc8db4ad7f02-kube-api-access-w2lks\") pod \"maas-api-e2e-autores-c8ee6b-6b8b548978-qt994\" (UID: \"744caa65-978b-4a72-af47-fc8db4ad7f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" Nov 03 02:46:47.196181 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:47.196104 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/744caa65-978b-4a72-af47-fc8db4ad7f02-maas-api-tls\") pod \"maas-api-e2e-autores-c8ee6b-6b8b548978-qt994\" (UID: \"744caa65-978b-4a72-af47-fc8db4ad7f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" Nov 03 02:46:47.196282 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:46:47.196265 5463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-serving-cert: secret "maas-api-e2e-autores-c8ee6b-serving-cert" not found Nov 03 02:46:47.196368 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:46:47.196358 5463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/744caa65-978b-4a72-af47-fc8db4ad7f02-maas-api-tls podName:744caa65-978b-4a72-af47-fc8db4ad7f02 nodeName:}" failed. No retries permitted until 2026-11-03 02:46:47.696337453 +0000 UTC m=+3289.380446607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/744caa65-978b-4a72-af47-fc8db4ad7f02-maas-api-tls") pod "maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" (UID: "744caa65-978b-4a72-af47-fc8db4ad7f02") : secret "maas-api-e2e-autores-c8ee6b-serving-cert" not found Nov 03 02:46:47.196638 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:47.196619 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/744caa65-978b-4a72-af47-fc8db4ad7f02-openshift-service-ca\") pod \"maas-api-e2e-autores-c8ee6b-6b8b548978-qt994\" (UID: \"744caa65-978b-4a72-af47-fc8db4ad7f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" Nov 03 02:46:47.198812 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:47.198788 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/744caa65-978b-4a72-af47-fc8db4ad7f02-metrics-certs\") pod \"maas-api-e2e-autores-c8ee6b-6b8b548978-qt994\" (UID: \"744caa65-978b-4a72-af47-fc8db4ad7f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" Nov 03 02:46:47.256771 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:47.256694 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2lks\" (UniqueName: \"kubernetes.io/projected/744caa65-978b-4a72-af47-fc8db4ad7f02-kube-api-access-w2lks\") pod \"maas-api-e2e-autores-c8ee6b-6b8b548978-qt994\" (UID: \"744caa65-978b-4a72-af47-fc8db4ad7f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" Nov 03 02:46:47.327918 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:47.327884 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:47.480577 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:47.480539 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:47.702342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:47.702305 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/744caa65-978b-4a72-af47-fc8db4ad7f02-maas-api-tls\") pod \"maas-api-e2e-autores-c8ee6b-6b8b548978-qt994\" (UID: \"744caa65-978b-4a72-af47-fc8db4ad7f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" Nov 03 02:46:47.704938 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:47.704915 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/744caa65-978b-4a72-af47-fc8db4ad7f02-maas-api-tls\") pod \"maas-api-e2e-autores-c8ee6b-6b8b548978-qt994\" (UID: \"744caa65-978b-4a72-af47-fc8db4ad7f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" Nov 03 02:46:47.880358 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:47.880321 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" Nov 03 02:46:48.258168 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:46:48.258135 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744caa65_978b_4a72_af47_fc8db4ad7f02.slice/crio-bc78a2a3184c041c218ebed618a9c9e385f2564def19a9ef7d9e2605a7df7e78 WatchSource:0}: Error finding container bc78a2a3184c041c218ebed618a9c9e385f2564def19a9ef7d9e2605a7df7e78: Status 404 returned error can't find the container with id bc78a2a3184c041c218ebed618a9c9e385f2564def19a9ef7d9e2605a7df7e78 Nov 03 02:46:48.260828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:48.260802 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994"] Nov 03 02:46:49.140836 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:49.140798 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" event={"ID":"744caa65-978b-4a72-af47-fc8db4ad7f02","Type":"ContainerStarted","Data":"bc78a2a3184c041c218ebed618a9c9e385f2564def19a9ef7d9e2605a7df7e78"} Nov 03 02:46:49.801469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:49.801433 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:49.982577 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:49.982537 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:50.162237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:50.157696 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:50.192336 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:50.192297 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" event={"ID":"744caa65-978b-4a72-af47-fc8db4ad7f02","Type":"ContainerStarted","Data":"72bee3e9e44fd0a2db73451009282ff8026f6c3904677e657f7e51a79a435c01"} Nov 03 02:46:50.192553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:50.192457 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" Nov 03 02:46:50.310455 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:50.310406 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" podStartSLOduration=3.418387916 podStartE2EDuration="4.310392565s" totalImagesPullingTime="892.004649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:46:50.308246426 +0000 UTC m=+3291.992355581" watchObservedRunningTime="2026-11-03 02:46:50.310392565 +0000 UTC m=+3291.994501720" Nov 03 02:46:52.447298 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:52.447263 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:53.028617 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:53.028583 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:53.204760 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:53.204730 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:56.204793 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:56.204757 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" Nov 03 02:46:57.703377 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:57.703343 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:59.974227 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:46:59.974169 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:02.637839 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:02.637802 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:03.023856 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:03.023829 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:47:03.024123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:03.024106 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:47:07.344731 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:07.344691 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:12.062140 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:12.062103 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:13.008431 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:47:13.008398 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:47:16.806938 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:16.806900 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:21.479547 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:21.479508 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:21.630288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:21.630253 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:23.255432 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:23.255392 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:26.527363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:26.527329 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:30.623632 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:30.623596 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:31.294920 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:31.294827 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:35.868052 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:35.868009 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:36.037150 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:36.037108 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:36.190461 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:36.190427 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:40.781068 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:40.780987 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:43.008706 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:47:43.008671 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:47:45.523040 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:45.523005 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:50.169675 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:50.169632 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:50.334627 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:50.334593 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:50.480157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:50.480043 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:55.140504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:55.140467 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:59.933075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:47:59.933041 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:04.663867 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:04.663825 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:09.368457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:09.368381 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:11.463379 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:11.463321 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:13.009348 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:48:13.009303 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:48:14.071336 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:14.071303 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:16.307216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:16.307171 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg"] Nov 03 02:48:16.307681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:16.307369 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" podUID="2d2157b2-209f-4ed7-9aab-a79fbe023d82" containerName="istio-proxy" containerID="cri-o://7326da7a8920c6ba29d5fe4fd2dcaea9e1ac8dd7e3a2d9d30881f207c375a047" gracePeriod=30 Nov 03 02:48:16.501622 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:16.501592 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:18.267422 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.267383 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:48:18.267858 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.267502 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" podUID="2d2157b2-209f-4ed7-9aab-a79fbe023d82" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:48:18.504051 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.504007 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5"] Nov 03 02:48:18.508376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.508352 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" Nov 03 02:48:18.564837 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.564755 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5"] Nov 03 02:48:18.608712 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.608675 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-e3cc05-serving-cert\"" Nov 03 02:48:18.641527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.641484 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls56q\" (UniqueName: \"kubernetes.io/projected/6f9aa54f-d8cd-437f-b428-711f985d0eca-kube-api-access-ls56q\") pod \"maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5\" (UID: \"6f9aa54f-d8cd-437f-b428-711f985d0eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" Nov 03 02:48:18.641705 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.641641 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9aa54f-d8cd-437f-b428-711f985d0eca-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5\" (UID: \"6f9aa54f-d8cd-437f-b428-711f985d0eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" Nov 03 02:48:18.641826 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.641799 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f9aa54f-d8cd-437f-b428-711f985d0eca-metrics-certs\") pod \"maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5\" (UID: \"6f9aa54f-d8cd-437f-b428-711f985d0eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" Nov 03 02:48:18.641928 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.641913 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f9aa54f-d8cd-437f-b428-711f985d0eca-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5\" (UID: \"6f9aa54f-d8cd-437f-b428-711f985d0eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" Nov 03 02:48:18.706614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.706573 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:18.742872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.742841 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls56q\" (UniqueName: \"kubernetes.io/projected/6f9aa54f-d8cd-437f-b428-711f985d0eca-kube-api-access-ls56q\") pod \"maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5\" (UID: \"6f9aa54f-d8cd-437f-b428-711f985d0eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" Nov 03 02:48:18.743035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.742933 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9aa54f-d8cd-437f-b428-711f985d0eca-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5\" (UID: \"6f9aa54f-d8cd-437f-b428-711f985d0eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" Nov 03 02:48:18.743035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.743028 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f9aa54f-d8cd-437f-b428-711f985d0eca-metrics-certs\") pod \"maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5\" (UID: \"6f9aa54f-d8cd-437f-b428-711f985d0eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" Nov 03 02:48:18.743144 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.743105 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f9aa54f-d8cd-437f-b428-711f985d0eca-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5\" (UID: \"6f9aa54f-d8cd-437f-b428-711f985d0eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" Nov 03 02:48:18.743676 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.743645 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9aa54f-d8cd-437f-b428-711f985d0eca-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5\" (UID: \"6f9aa54f-d8cd-437f-b428-711f985d0eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" Nov 03 02:48:18.745457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.745427 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f9aa54f-d8cd-437f-b428-711f985d0eca-metrics-certs\") pod \"maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5\" (UID: \"6f9aa54f-d8cd-437f-b428-711f985d0eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" Nov 03 02:48:18.749606 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.749584 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f9aa54f-d8cd-437f-b428-711f985d0eca-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5\" (UID: \"6f9aa54f-d8cd-437f-b428-711f985d0eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" Nov 03 02:48:18.800672 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.800642 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls56q\" (UniqueName: \"kubernetes.io/projected/6f9aa54f-d8cd-437f-b428-711f985d0eca-kube-api-access-ls56q\") pod \"maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5\" (UID: \"6f9aa54f-d8cd-437f-b428-711f985d0eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" Nov 03 02:48:18.823815 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.823753 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" Nov 03 02:48:18.867615 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:18.867580 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:19.053960 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:19.053920 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:19.208047 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:48:19.208019 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9aa54f_d8cd_437f_b428_711f985d0eca.slice/crio-02503a2ac19617ec71ecfe79cb3e3c9284848d86daec76868e4750e662915f39 WatchSource:0}: Error finding container 02503a2ac19617ec71ecfe79cb3e3c9284848d86daec76868e4750e662915f39: Status 404 returned error can't find the container with id 02503a2ac19617ec71ecfe79cb3e3c9284848d86daec76868e4750e662915f39 Nov 03 02:48:19.210406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:19.210383 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5"] Nov 03 02:48:19.879331 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:19.879293 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" event={"ID":"6f9aa54f-d8cd-437f-b428-711f985d0eca","Type":"ContainerStarted","Data":"02503a2ac19617ec71ecfe79cb3e3c9284848d86daec76868e4750e662915f39"} Nov 03 02:48:20.888177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:20.888130 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" event={"ID":"6f9aa54f-d8cd-437f-b428-711f985d0eca","Type":"ContainerStarted","Data":"e949b231ac5d0722f66438487002adc2e8f6f1cf244fd238ba1fae98035f71a2"} Nov 03 02:48:20.888717 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:20.888212 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" Nov 03 02:48:21.009368 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.005353 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" podStartSLOduration=2.119171588 podStartE2EDuration="3.005337847s" totalImagesPullingTime="886.166259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:48:21.003697204 +0000 UTC m=+3382.687806363" watchObservedRunningTime="2026-11-03 02:48:21.005337847 +0000 UTC m=+3382.689447002" Nov 03 02:48:21.581339 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.581268 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:48:21.682111 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.682078 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2157b2-209f-4ed7-9aab-a79fbe023d82-kube-api-access-2slns\" (UniqueName: \"kubernetes.io/projected/2d2157b2-209f-4ed7-9aab-a79fbe023d82-kube-api-access-2slns\") pod \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " Nov 03 02:48:21.682342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.682222 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-data\") pod \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " Nov 03 02:48:21.682342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.682266 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-workload-certs\") pod \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " Nov 03 02:48:21.682342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.682298 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-envoy\") pod \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " Nov 03 02:48:21.682342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.682324 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-token\" (UniqueName: \"kubernetes.io/projected/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-token\") pod \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " Nov 03 02:48:21.682599 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.682364 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-workload-socket\") pod \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " Nov 03 02:48:21.682599 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.682411 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-podinfo\") pod \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " Nov 03 02:48:21.682599 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.682472 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-credential-socket\") pod \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " Nov 03 02:48:21.682599 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.682520 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istiod-ca-cert\") pod \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\" (UID: \"2d2157b2-209f-4ed7-9aab-a79fbe023d82\") " Nov 03 02:48:21.682599 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.682549 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-data" pod "2d2157b2-209f-4ed7-9aab-a79fbe023d82" (UID: "2d2157b2-209f-4ed7-9aab-a79fbe023d82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:48:21.682906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.682641 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-workload-certs" pod "2d2157b2-209f-4ed7-9aab-a79fbe023d82" (UID: "2d2157b2-209f-4ed7-9aab-a79fbe023d82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:48:21.682906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.682792 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-workload-socket" pod "2d2157b2-209f-4ed7-9aab-a79fbe023d82" (UID: "2d2157b2-209f-4ed7-9aab-a79fbe023d82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:48:21.682906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.682841 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-credential-socket" pod "2d2157b2-209f-4ed7-9aab-a79fbe023d82" (UID: "2d2157b2-209f-4ed7-9aab-a79fbe023d82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:48:21.683102 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.683082 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:21.683158 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.683125 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:21.683158 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.683139 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:21.683158 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.683153 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:21.683365 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.683089 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istiod-ca-cert" pod "2d2157b2-209f-4ed7-9aab-a79fbe023d82" (UID: "2d2157b2-209f-4ed7-9aab-a79fbe023d82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:48:21.684834 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.684806 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-token" pod "2d2157b2-209f-4ed7-9aab-a79fbe023d82" (UID: "2d2157b2-209f-4ed7-9aab-a79fbe023d82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:48:21.685035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.685012 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-envoy" pod "2d2157b2-209f-4ed7-9aab-a79fbe023d82" (UID: "2d2157b2-209f-4ed7-9aab-a79fbe023d82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:48:21.685126 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.685101 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-podinfo" pod "2d2157b2-209f-4ed7-9aab-a79fbe023d82" (UID: "2d2157b2-209f-4ed7-9aab-a79fbe023d82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:48:21.685425 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.685410 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2157b2-209f-4ed7-9aab-a79fbe023d82-kube-api-access-2slns" pod "2d2157b2-209f-4ed7-9aab-a79fbe023d82" (UID: "2d2157b2-209f-4ed7-9aab-a79fbe023d82"). InnerVolumeSpecName "kube-api-access-2slns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:48:21.769478 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.769348 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:21.784186 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.784154 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:21.784186 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.784183 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:21.784415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.784213 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:21.784415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.784223 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2157b2-209f-4ed7-9aab-a79fbe023d82-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:21.784415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.784232 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2slns\" (UniqueName: \"kubernetes.io/projected/2d2157b2-209f-4ed7-9aab-a79fbe023d82-kube-api-access-2slns\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:21.898334 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.898300 5463 generic.go:356] "Generic (PLEG): container finished" podID="2d2157b2-209f-4ed7-9aab-a79fbe023d82" containerID="7326da7a8920c6ba29d5fe4fd2dcaea9e1ac8dd7e3a2d9d30881f207c375a047" exitCode=0 Nov 03 02:48:21.898804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.898362 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" Nov 03 02:48:21.898804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.898380 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" event={"ID":"2d2157b2-209f-4ed7-9aab-a79fbe023d82","Type":"ContainerDied","Data":"7326da7a8920c6ba29d5fe4fd2dcaea9e1ac8dd7e3a2d9d30881f207c375a047"} Nov 03 02:48:21.898804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.898459 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg" event={"ID":"2d2157b2-209f-4ed7-9aab-a79fbe023d82","Type":"ContainerDied","Data":"3a085d3d9315b5ef6b1e4372dcd0cbb1597f06efec3087ed46b6291b6b23abbe"} Nov 03 02:48:21.898804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.898481 5463 scope.go:122] "RemoveContainer" containerID="7326da7a8920c6ba29d5fe4fd2dcaea9e1ac8dd7e3a2d9d30881f207c375a047" Nov 03 02:48:21.915007 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.914986 5463 scope.go:122] "RemoveContainer" containerID="92bf2fd408a99e0a6e1de092dee3d681167da3227d30119d33c6a565c74042b6" Nov 03 02:48:21.925588 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.925568 5463 scope.go:122] "RemoveContainer" containerID="7326da7a8920c6ba29d5fe4fd2dcaea9e1ac8dd7e3a2d9d30881f207c375a047" Nov 03 02:48:21.925843 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:48:21.925822 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7326da7a8920c6ba29d5fe4fd2dcaea9e1ac8dd7e3a2d9d30881f207c375a047\": container with ID starting with 7326da7a8920c6ba29d5fe4fd2dcaea9e1ac8dd7e3a2d9d30881f207c375a047 not found: ID does not exist" containerID="7326da7a8920c6ba29d5fe4fd2dcaea9e1ac8dd7e3a2d9d30881f207c375a047" Nov 03 02:48:21.925902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.925854 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7326da7a8920c6ba29d5fe4fd2dcaea9e1ac8dd7e3a2d9d30881f207c375a047"} err="failed to get container status \"7326da7a8920c6ba29d5fe4fd2dcaea9e1ac8dd7e3a2d9d30881f207c375a047\": rpc error: code = NotFound desc = could not find container \"7326da7a8920c6ba29d5fe4fd2dcaea9e1ac8dd7e3a2d9d30881f207c375a047\": container with ID starting with 7326da7a8920c6ba29d5fe4fd2dcaea9e1ac8dd7e3a2d9d30881f207c375a047 not found: ID does not exist" Nov 03 02:48:21.925902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.925870 5463 scope.go:122] "RemoveContainer" containerID="92bf2fd408a99e0a6e1de092dee3d681167da3227d30119d33c6a565c74042b6" Nov 03 02:48:21.926180 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:48:21.926154 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92bf2fd408a99e0a6e1de092dee3d681167da3227d30119d33c6a565c74042b6\": container with ID starting with 92bf2fd408a99e0a6e1de092dee3d681167da3227d30119d33c6a565c74042b6 not found: ID does not exist" containerID="92bf2fd408a99e0a6e1de092dee3d681167da3227d30119d33c6a565c74042b6" Nov 03 02:48:21.926265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:21.926186 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92bf2fd408a99e0a6e1de092dee3d681167da3227d30119d33c6a565c74042b6"} err="failed to get container status \"92bf2fd408a99e0a6e1de092dee3d681167da3227d30119d33c6a565c74042b6\": rpc error: code = NotFound desc = could not find container \"92bf2fd408a99e0a6e1de092dee3d681167da3227d30119d33c6a565c74042b6\": container with ID starting with 92bf2fd408a99e0a6e1de092dee3d681167da3227d30119d33c6a565c74042b6 not found: ID does not exist" Nov 03 02:48:22.065734 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:22.065706 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg"] Nov 03 02:48:22.068648 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:22.068626 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cf4pg"] Nov 03 02:48:22.446041 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:22.446010 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2157b2-209f-4ed7-9aab-a79fbe023d82" path="/var/lib/kubelet/pods/2d2157b2-209f-4ed7-9aab-a79fbe023d82/volumes" Nov 03 02:48:23.204395 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:23.204367 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:24.038055 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:24.038016 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:24.944956 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:24.944917 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:26.904638 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:26.904608 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" Nov 03 02:48:27.338563 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:27.338479 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:27.749631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:27.749591 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:27.907608 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:27.907572 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:28.807286 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:28.807256 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:33.523702 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:33.523660 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:38.238650 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:38.238610 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:43.010273 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:48:43.010234 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:48:43.022903 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:43.022881 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:47.836103 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:47.836050 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:52.578788 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:52.578749 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:53.216993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:53.216938 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:53.363304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:53.363269 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:53.733707 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:53.733674 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:57.582918 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:57.582875 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:59.206206 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:48:59.206166 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:02.621334 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:02.621274 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:07.330555 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:07.330523 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:12.018828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:12.018745 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:13.008861 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:49:13.008829 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:49:16.938122 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:16.938085 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:17.179103 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:17.179066 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:18.853859 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:18.853820 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:22.086304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:22.086269 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:22.242878 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:22.242841 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:22.744089 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:22.744051 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:23.193095 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:23.193060 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:24.148320 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:24.148275 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:24.305672 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:24.305632 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:27.133979 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:27.133936 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:29.074507 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:29.074461 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:29.237238 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:29.237180 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:32.105890 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:32.105856 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:33.837282 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:33.837252 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:34.327938 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:34.327879 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:35.111518 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:35.111478 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:37.210346 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:37.210315 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:39.455200 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:39.455134 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:42.348314 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:42.348282 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:43.008288 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:49:43.008252 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:49:47.552681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:47.552610 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:49.119845 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:49.119810 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:52.572837 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:52.572796 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:54.760235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:54.759125 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:54.915815 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:54.915779 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:55.182766 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:55.182668 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:57.192516 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:49:57.192474 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:01.018123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:01.018087 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:01.985053 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:01.985013 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:06.697695 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:06.697660 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:11.412322 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:11.412279 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:13.007954 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:50:13.007919 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:50:16.160978 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:16.160937 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:16.315874 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:16.315845 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:16.978339 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:16.978305 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc"] Nov 03 02:50:17.085772 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:17.085739 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g"] Nov 03 02:50:17.346366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:17.346283 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:17.500093 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:17.500032 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:20.688613 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:20.688549 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l"] Nov 03 02:50:20.689074 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:20.688810 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" podUID="66888dec-a1ab-4541-9c61-58014ece2263" containerName="maas-api" containerID="cri-o://af9608a7a1ad615aad185814a3371019813ef0d5dc134ab3b7f250e178b25d10" gracePeriod=30 Nov 03 02:50:20.837458 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:20.837434 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:20.867571 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:20.867535 5463 generic.go:356] "Generic (PLEG): container finished" podID="66888dec-a1ab-4541-9c61-58014ece2263" containerID="af9608a7a1ad615aad185814a3371019813ef0d5dc134ab3b7f250e178b25d10" exitCode=0 Nov 03 02:50:20.867737 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:20.867596 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" event={"ID":"66888dec-a1ab-4541-9c61-58014ece2263","Type":"ContainerDied","Data":"af9608a7a1ad615aad185814a3371019813ef0d5dc134ab3b7f250e178b25d10"} Nov 03 02:50:20.983963 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:20.983934 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" Nov 03 02:50:21.154176 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:21.154139 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66888dec-a1ab-4541-9c61-58014ece2263-kube-api-access-2nk64\" (UniqueName: \"kubernetes.io/projected/66888dec-a1ab-4541-9c61-58014ece2263-kube-api-access-2nk64\") pod \"66888dec-a1ab-4541-9c61-58014ece2263\" (UID: \"66888dec-a1ab-4541-9c61-58014ece2263\") " Nov 03 02:50:21.154378 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:21.154215 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66888dec-a1ab-4541-9c61-58014ece2263-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66888dec-a1ab-4541-9c61-58014ece2263-maas-api-tls\") pod \"66888dec-a1ab-4541-9c61-58014ece2263\" (UID: \"66888dec-a1ab-4541-9c61-58014ece2263\") " Nov 03 02:50:21.154378 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:21.154327 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66888dec-a1ab-4541-9c61-58014ece2263-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66888dec-a1ab-4541-9c61-58014ece2263-openshift-service-ca\") pod \"66888dec-a1ab-4541-9c61-58014ece2263\" (UID: \"66888dec-a1ab-4541-9c61-58014ece2263\") " Nov 03 02:50:21.154529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:21.154428 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66888dec-a1ab-4541-9c61-58014ece2263-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66888dec-a1ab-4541-9c61-58014ece2263-metrics-certs\") pod \"66888dec-a1ab-4541-9c61-58014ece2263\" (UID: \"66888dec-a1ab-4541-9c61-58014ece2263\") " Nov 03 02:50:21.154740 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:21.154710 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66888dec-a1ab-4541-9c61-58014ece2263-openshift-service-ca" pod "66888dec-a1ab-4541-9c61-58014ece2263" (UID: "66888dec-a1ab-4541-9c61-58014ece2263"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:50:21.154960 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:21.154939 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66888dec-a1ab-4541-9c61-58014ece2263-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:21.156404 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:21.156372 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66888dec-a1ab-4541-9c61-58014ece2263-kube-api-access-2nk64" pod "66888dec-a1ab-4541-9c61-58014ece2263" (UID: "66888dec-a1ab-4541-9c61-58014ece2263"). InnerVolumeSpecName "kube-api-access-2nk64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:50:21.156515 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:21.156465 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66888dec-a1ab-4541-9c61-58014ece2263-metrics-certs" pod "66888dec-a1ab-4541-9c61-58014ece2263" (UID: "66888dec-a1ab-4541-9c61-58014ece2263"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:50:21.156561 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:21.156516 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66888dec-a1ab-4541-9c61-58014ece2263-maas-api-tls" pod "66888dec-a1ab-4541-9c61-58014ece2263" (UID: "66888dec-a1ab-4541-9c61-58014ece2263"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:50:21.256634 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:21.256550 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66888dec-a1ab-4541-9c61-58014ece2263-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:21.256634 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:21.256587 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nk64\" (UniqueName: \"kubernetes.io/projected/66888dec-a1ab-4541-9c61-58014ece2263-kube-api-access-2nk64\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:21.256634 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:21.256603 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66888dec-a1ab-4541-9c61-58014ece2263-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:21.738794 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:21.738267 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:21.876614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:21.876580 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" Nov 03 02:50:21.876792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:21.876611 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l" event={"ID":"66888dec-a1ab-4541-9c61-58014ece2263","Type":"ContainerDied","Data":"d2ce19631b77071fbd068e67d52e849b29fecb0beeb86f178488418517fa959c"} Nov 03 02:50:21.876792 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:21.876676 5463 scope.go:122] "RemoveContainer" containerID="af9608a7a1ad615aad185814a3371019813ef0d5dc134ab3b7f250e178b25d10" Nov 03 02:50:21.978728 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:21.978676 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc" podUID="5ec0819e-4232-43a9-820e-3a7b65b7ac9d" containerName="payload-processing" containerID="cri-o://a705845d5a90b977261816a518e7d615358e266c9acc967852e39dfc5c363a5c" gracePeriod=25 Nov 03 02:50:22.041238 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.041180 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l"] Nov 03 02:50:22.086486 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.086382 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g" podUID="64bcbb35-8a22-43a4-beae-fbd0420e5bdd" containerName="payload-pre-processing" containerID="cri-o://762fa72ffa6f5caa29664846acee6f3be0ff27961b236490f7ec660476c2b7f6" gracePeriod=25 Nov 03 02:50:22.092271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.092181 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-038c21-6479f66779-drf7l"] Nov 03 02:50:22.261698 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.261634 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc" Nov 03 02:50:22.371863 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.370880 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ec0819e-4232-43a9-820e-3a7b65b7ac9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec0819e-4232-43a9-820e-3a7b65b7ac9d-plugins-config-volume\") pod \"5ec0819e-4232-43a9-820e-3a7b65b7ac9d\" (UID: \"5ec0819e-4232-43a9-820e-3a7b65b7ac9d\") " Nov 03 02:50:22.371863 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.371057 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec0819e-4232-43a9-820e-3a7b65b7ac9d-kube-api-access-p6rfp\" (UniqueName: \"kubernetes.io/projected/5ec0819e-4232-43a9-820e-3a7b65b7ac9d-kube-api-access-p6rfp\") pod \"5ec0819e-4232-43a9-820e-3a7b65b7ac9d\" (UID: \"5ec0819e-4232-43a9-820e-3a7b65b7ac9d\") " Nov 03 02:50:22.371863 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.371510 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec0819e-4232-43a9-820e-3a7b65b7ac9d-plugins-config-volume" pod "5ec0819e-4232-43a9-820e-3a7b65b7ac9d" (UID: "5ec0819e-4232-43a9-820e-3a7b65b7ac9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:50:22.371863 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.371631 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec0819e-4232-43a9-820e-3a7b65b7ac9d-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:22.376313 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.374890 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec0819e-4232-43a9-820e-3a7b65b7ac9d-kube-api-access-p6rfp" pod "5ec0819e-4232-43a9-820e-3a7b65b7ac9d" (UID: "5ec0819e-4232-43a9-820e-3a7b65b7ac9d"). InnerVolumeSpecName "kube-api-access-p6rfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:50:22.383968 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.383946 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g" Nov 03 02:50:22.399808 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.399780 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:22.445344 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.445318 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66888dec-a1ab-4541-9c61-58014ece2263" path="/var/lib/kubelet/pods/66888dec-a1ab-4541-9c61-58014ece2263/volumes" Nov 03 02:50:22.472938 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.472914 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64bcbb35-8a22-43a4-beae-fbd0420e5bdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64bcbb35-8a22-43a4-beae-fbd0420e5bdd-plugins-config-volume\") pod \"64bcbb35-8a22-43a4-beae-fbd0420e5bdd\" (UID: \"64bcbb35-8a22-43a4-beae-fbd0420e5bdd\") " Nov 03 02:50:22.473071 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.472965 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64bcbb35-8a22-43a4-beae-fbd0420e5bdd-kube-api-access-lplm6\" (UniqueName: \"kubernetes.io/projected/64bcbb35-8a22-43a4-beae-fbd0420e5bdd-kube-api-access-lplm6\") pod \"64bcbb35-8a22-43a4-beae-fbd0420e5bdd\" (UID: \"64bcbb35-8a22-43a4-beae-fbd0420e5bdd\") " Nov 03 02:50:22.473330 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.473305 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64bcbb35-8a22-43a4-beae-fbd0420e5bdd-plugins-config-volume" pod "64bcbb35-8a22-43a4-beae-fbd0420e5bdd" (UID: "64bcbb35-8a22-43a4-beae-fbd0420e5bdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:50:22.473447 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.473433 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64bcbb35-8a22-43a4-beae-fbd0420e5bdd-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:22.473494 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.473450 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6rfp\" (UniqueName: \"kubernetes.io/projected/5ec0819e-4232-43a9-820e-3a7b65b7ac9d-kube-api-access-p6rfp\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:22.475334 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.475312 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64bcbb35-8a22-43a4-beae-fbd0420e5bdd-kube-api-access-lplm6" pod "64bcbb35-8a22-43a4-beae-fbd0420e5bdd" (UID: "64bcbb35-8a22-43a4-beae-fbd0420e5bdd"). InnerVolumeSpecName "kube-api-access-lplm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:50:22.572028 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.571949 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:22.574470 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.574450 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lplm6\" (UniqueName: \"kubernetes.io/projected/64bcbb35-8a22-43a4-beae-fbd0420e5bdd-kube-api-access-lplm6\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:22.730454 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.730422 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:22.889150 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.889063 5463 generic.go:356] "Generic (PLEG): container finished" podID="5ec0819e-4232-43a9-820e-3a7b65b7ac9d" containerID="a705845d5a90b977261816a518e7d615358e266c9acc967852e39dfc5c363a5c" exitCode=0 Nov 03 02:50:22.889150 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.889112 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc" Nov 03 02:50:22.889734 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.889154 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc" event={"ID":"5ec0819e-4232-43a9-820e-3a7b65b7ac9d","Type":"ContainerDied","Data":"a705845d5a90b977261816a518e7d615358e266c9acc967852e39dfc5c363a5c"} Nov 03 02:50:22.889734 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.889219 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc" event={"ID":"5ec0819e-4232-43a9-820e-3a7b65b7ac9d","Type":"ContainerDied","Data":"e35aa638fc1ca9bcb67b1b0152e83afa99452ce73380b7b86d855578fc06fbb9"} Nov 03 02:50:22.889734 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.889245 5463 scope.go:122] "RemoveContainer" containerID="a705845d5a90b977261816a518e7d615358e266c9acc967852e39dfc5c363a5c" Nov 03 02:50:22.890866 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.890841 5463 generic.go:356] "Generic (PLEG): container finished" podID="64bcbb35-8a22-43a4-beae-fbd0420e5bdd" containerID="762fa72ffa6f5caa29664846acee6f3be0ff27961b236490f7ec660476c2b7f6" exitCode=0 Nov 03 02:50:22.891021 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.890887 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g" Nov 03 02:50:22.891021 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.890896 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g" event={"ID":"64bcbb35-8a22-43a4-beae-fbd0420e5bdd","Type":"ContainerDied","Data":"762fa72ffa6f5caa29664846acee6f3be0ff27961b236490f7ec660476c2b7f6"} Nov 03 02:50:22.891021 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.890932 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g" event={"ID":"64bcbb35-8a22-43a4-beae-fbd0420e5bdd","Type":"ContainerDied","Data":"36ccfb53fd5b5e004c01be16ef80c01818c2b8036f7a8691aacf06cba42420c7"} Nov 03 02:50:22.901573 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.901555 5463 scope.go:122] "RemoveContainer" containerID="a705845d5a90b977261816a518e7d615358e266c9acc967852e39dfc5c363a5c" Nov 03 02:50:22.901858 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:50:22.901837 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a705845d5a90b977261816a518e7d615358e266c9acc967852e39dfc5c363a5c\": container with ID starting with a705845d5a90b977261816a518e7d615358e266c9acc967852e39dfc5c363a5c not found: ID does not exist" containerID="a705845d5a90b977261816a518e7d615358e266c9acc967852e39dfc5c363a5c" Nov 03 02:50:22.901942 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.901868 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a705845d5a90b977261816a518e7d615358e266c9acc967852e39dfc5c363a5c"} err="failed to get container status \"a705845d5a90b977261816a518e7d615358e266c9acc967852e39dfc5c363a5c\": rpc error: code = NotFound desc = could not find container \"a705845d5a90b977261816a518e7d615358e266c9acc967852e39dfc5c363a5c\": container with ID starting with a705845d5a90b977261816a518e7d615358e266c9acc967852e39dfc5c363a5c not found: ID does not exist" Nov 03 02:50:22.901942 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.901888 5463 scope.go:122] "RemoveContainer" containerID="762fa72ffa6f5caa29664846acee6f3be0ff27961b236490f7ec660476c2b7f6" Nov 03 02:50:22.913317 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.913300 5463 scope.go:122] "RemoveContainer" containerID="762fa72ffa6f5caa29664846acee6f3be0ff27961b236490f7ec660476c2b7f6" Nov 03 02:50:22.913579 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:50:22.913558 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"762fa72ffa6f5caa29664846acee6f3be0ff27961b236490f7ec660476c2b7f6\": container with ID starting with 762fa72ffa6f5caa29664846acee6f3be0ff27961b236490f7ec660476c2b7f6 not found: ID does not exist" containerID="762fa72ffa6f5caa29664846acee6f3be0ff27961b236490f7ec660476c2b7f6" Nov 03 02:50:22.913631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:22.913586 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762fa72ffa6f5caa29664846acee6f3be0ff27961b236490f7ec660476c2b7f6"} err="failed to get container status \"762fa72ffa6f5caa29664846acee6f3be0ff27961b236490f7ec660476c2b7f6\": rpc error: code = NotFound desc = could not find container \"762fa72ffa6f5caa29664846acee6f3be0ff27961b236490f7ec660476c2b7f6\": container with ID starting with 762fa72ffa6f5caa29664846acee6f3be0ff27961b236490f7ec660476c2b7f6 not found: ID does not exist" Nov 03 02:50:23.056543 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:23.056509 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g"] Nov 03 02:50:23.061216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:23.061172 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c8ee6b-794c67d5c5-kqq6g"] Nov 03 02:50:23.167292 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:23.167257 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc"] Nov 03 02:50:23.170164 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:23.170136 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c8ee6b-7fd6976c9f-lrsjc"] Nov 03 02:50:23.240625 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:23.240597 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:24.442970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:24.442936 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ec0819e-4232-43a9-820e-3a7b65b7ac9d" path="/var/lib/kubelet/pods/5ec0819e-4232-43a9-820e-3a7b65b7ac9d/volumes" Nov 03 02:50:24.443697 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:24.443674 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64bcbb35-8a22-43a4-beae-fbd0420e5bdd" path="/var/lib/kubelet/pods/64bcbb35-8a22-43a4-beae-fbd0420e5bdd/volumes" Nov 03 02:50:25.264845 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:25.264800 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:25.867595 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:25.867564 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:26.284676 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:26.284644 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:27.855238 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:27.855185 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:30.494061 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:30.494023 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:30.655594 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:30.655552 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:30.819329 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:30.819237 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:31.313535 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:31.313503 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:31.477862 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:31.477830 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:33.038781 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:33.038749 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:33.998282 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:33.998234 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:34.188130 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:34.188103 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:37.408512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:37.408479 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:39.114242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:39.114155 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:43.008350 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:50:43.008315 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:50:43.805550 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:43.805515 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:44.926683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:44.926649 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:45.133480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:45.133451 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:46.482616 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:46.482580 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:47.064652 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:47.064622 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:47.379369 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:47.379268 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m"] Nov 03 02:50:47.379570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:47.379522 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" podUID="da50d3ef-a1d7-4546-8e52-60ee075543c3" containerName="istio-proxy" containerID="cri-o://db84957e8b9cab83d1c05c1b12c15ccc42f959dd0bb276a8bfe3e5ac0dceecf2" gracePeriod=30 Nov 03 02:50:48.200283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:48.200247 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:49.045885 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:49.045849 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:50:49.046065 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:49.045918 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" podUID="da50d3ef-a1d7-4546-8e52-60ee075543c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:50:50.171493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:50.171457 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:50.786772 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:50.786739 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:50.949019 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:50.948952 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:52.239247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:52.239182 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:52.738271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:52.738239 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:53.049915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.049889 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:50:53.143983 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.143954 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-token\" (UniqueName: \"kubernetes.io/projected/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-token\") pod \"da50d3ef-a1d7-4546-8e52-60ee075543c3\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " Nov 03 02:50:53.144160 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.144014 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-credential-socket\") pod \"da50d3ef-a1d7-4546-8e52-60ee075543c3\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " Nov 03 02:50:53.144160 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.144055 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da50d3ef-a1d7-4546-8e52-60ee075543c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da50d3ef-a1d7-4546-8e52-60ee075543c3-istiod-ca-cert\") pod \"da50d3ef-a1d7-4546-8e52-60ee075543c3\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " Nov 03 02:50:53.144160 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.144097 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da50d3ef-a1d7-4546-8e52-60ee075543c3-kube-api-access-tbsvs\" (UniqueName: \"kubernetes.io/projected/da50d3ef-a1d7-4546-8e52-60ee075543c3-kube-api-access-tbsvs\") pod \"da50d3ef-a1d7-4546-8e52-60ee075543c3\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " Nov 03 02:50:53.144376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.144174 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-data\") pod \"da50d3ef-a1d7-4546-8e52-60ee075543c3\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " Nov 03 02:50:53.144376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.144288 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-podinfo\") pod \"da50d3ef-a1d7-4546-8e52-60ee075543c3\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " Nov 03 02:50:53.144376 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.144332 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-envoy\") pod \"da50d3ef-a1d7-4546-8e52-60ee075543c3\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " Nov 03 02:50:53.144551 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.144461 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-workload-socket\") pod \"da50d3ef-a1d7-4546-8e52-60ee075543c3\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " Nov 03 02:50:53.144551 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.144512 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-workload-certs\") pod \"da50d3ef-a1d7-4546-8e52-60ee075543c3\" (UID: \"da50d3ef-a1d7-4546-8e52-60ee075543c3\") " Nov 03 02:50:53.144551 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.144530 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-data" pod "da50d3ef-a1d7-4546-8e52-60ee075543c3" (UID: "da50d3ef-a1d7-4546-8e52-60ee075543c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:50:53.144551 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.144538 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da50d3ef-a1d7-4546-8e52-60ee075543c3-istiod-ca-cert" pod "da50d3ef-a1d7-4546-8e52-60ee075543c3" (UID: "da50d3ef-a1d7-4546-8e52-60ee075543c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:50:53.144799 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.144775 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-credential-socket" pod "da50d3ef-a1d7-4546-8e52-60ee075543c3" (UID: "da50d3ef-a1d7-4546-8e52-60ee075543c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:50:53.144799 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.144786 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-workload-socket" pod "da50d3ef-a1d7-4546-8e52-60ee075543c3" (UID: "da50d3ef-a1d7-4546-8e52-60ee075543c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:50:53.144942 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.144923 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-workload-certs" pod "da50d3ef-a1d7-4546-8e52-60ee075543c3" (UID: "da50d3ef-a1d7-4546-8e52-60ee075543c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:50:53.145133 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.145109 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:53.145133 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.145132 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:53.145386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.145142 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:53.145386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.145151 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:53.145386 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.145159 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da50d3ef-a1d7-4546-8e52-60ee075543c3-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:53.146692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.146635 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-token" pod "da50d3ef-a1d7-4546-8e52-60ee075543c3" (UID: "da50d3ef-a1d7-4546-8e52-60ee075543c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:50:53.146987 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.146962 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-podinfo" pod "da50d3ef-a1d7-4546-8e52-60ee075543c3" (UID: "da50d3ef-a1d7-4546-8e52-60ee075543c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:50:53.147145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.147123 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da50d3ef-a1d7-4546-8e52-60ee075543c3-kube-api-access-tbsvs" pod "da50d3ef-a1d7-4546-8e52-60ee075543c3" (UID: "da50d3ef-a1d7-4546-8e52-60ee075543c3"). InnerVolumeSpecName "kube-api-access-tbsvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:50:53.147494 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.147470 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-envoy" pod "da50d3ef-a1d7-4546-8e52-60ee075543c3" (UID: "da50d3ef-a1d7-4546-8e52-60ee075543c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:50:53.217567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.217527 5463 generic.go:356] "Generic (PLEG): container finished" podID="da50d3ef-a1d7-4546-8e52-60ee075543c3" containerID="db84957e8b9cab83d1c05c1b12c15ccc42f959dd0bb276a8bfe3e5ac0dceecf2" exitCode=0 Nov 03 02:50:53.217730 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.217650 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" event={"ID":"da50d3ef-a1d7-4546-8e52-60ee075543c3","Type":"ContainerDied","Data":"db84957e8b9cab83d1c05c1b12c15ccc42f959dd0bb276a8bfe3e5ac0dceecf2"} Nov 03 02:50:53.217828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.217808 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" event={"ID":"da50d3ef-a1d7-4546-8e52-60ee075543c3","Type":"ContainerDied","Data":"d09b326ee8c63cebb229a416543f949ada75640ae5434f6dd298e142f5e9858a"} Nov 03 02:50:53.217892 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.217874 5463 scope.go:122] "RemoveContainer" containerID="db84957e8b9cab83d1c05c1b12c15ccc42f959dd0bb276a8bfe3e5ac0dceecf2" Nov 03 02:50:53.218468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.218444 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m" Nov 03 02:50:53.233529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.231449 5463 scope.go:122] "RemoveContainer" containerID="db84957e8b9cab83d1c05c1b12c15ccc42f959dd0bb276a8bfe3e5ac0dceecf2" Nov 03 02:50:53.233529 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:50:53.231840 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db84957e8b9cab83d1c05c1b12c15ccc42f959dd0bb276a8bfe3e5ac0dceecf2\": container with ID starting with db84957e8b9cab83d1c05c1b12c15ccc42f959dd0bb276a8bfe3e5ac0dceecf2 not found: ID does not exist" containerID="db84957e8b9cab83d1c05c1b12c15ccc42f959dd0bb276a8bfe3e5ac0dceecf2" Nov 03 02:50:53.233529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.231878 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db84957e8b9cab83d1c05c1b12c15ccc42f959dd0bb276a8bfe3e5ac0dceecf2"} err="failed to get container status \"db84957e8b9cab83d1c05c1b12c15ccc42f959dd0bb276a8bfe3e5ac0dceecf2\": rpc error: code = NotFound desc = could not find container \"db84957e8b9cab83d1c05c1b12c15ccc42f959dd0bb276a8bfe3e5ac0dceecf2\": container with ID starting with db84957e8b9cab83d1c05c1b12c15ccc42f959dd0bb276a8bfe3e5ac0dceecf2 not found: ID does not exist" Nov 03 02:50:53.246469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.246444 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:53.246469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.246470 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbsvs\" (UniqueName: \"kubernetes.io/projected/da50d3ef-a1d7-4546-8e52-60ee075543c3-kube-api-access-tbsvs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:53.246902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.246481 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:53.246902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.246489 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da50d3ef-a1d7-4546-8e52-60ee075543c3-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:50:53.385683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.385529 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m"] Nov 03 02:50:53.388427 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.388397 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-tm88m"] Nov 03 02:50:53.640740 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:53.640642 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:54.444417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:54.444386 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da50d3ef-a1d7-4546-8e52-60ee075543c3" path="/var/lib/kubelet/pods/da50d3ef-a1d7-4546-8e52-60ee075543c3/volumes" Nov 03 02:50:55.383237 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:55.383178 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:56.008272 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:56.008231 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:56.163146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:50:56.163117 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:00.333327 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:00.333293 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:05.058448 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:05.058409 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:09.156986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:09.156941 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5"] Nov 03 02:51:09.157489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:09.157187 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" podUID="6f9aa54f-d8cd-437f-b428-711f985d0eca" containerName="maas-api" containerID="cri-o://e949b231ac5d0722f66438487002adc2e8f6f1cf244fd238ba1fae98035f71a2" gracePeriod=30 Nov 03 02:51:09.279915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:09.279877 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:09.340441 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:09.340412 5463 generic.go:356] "Generic (PLEG): container finished" podID="6f9aa54f-d8cd-437f-b428-711f985d0eca" containerID="e949b231ac5d0722f66438487002adc2e8f6f1cf244fd238ba1fae98035f71a2" exitCode=0 Nov 03 02:51:09.340634 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:09.340482 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" event={"ID":"6f9aa54f-d8cd-437f-b428-711f985d0eca","Type":"ContainerDied","Data":"e949b231ac5d0722f66438487002adc2e8f6f1cf244fd238ba1fae98035f71a2"} Nov 03 02:51:09.583263 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:09.583236 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" Nov 03 02:51:09.689241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:09.689170 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f9aa54f-d8cd-437f-b428-711f985d0eca-kube-api-access-ls56q\" (UniqueName: \"kubernetes.io/projected/6f9aa54f-d8cd-437f-b428-711f985d0eca-kube-api-access-ls56q\") pod \"6f9aa54f-d8cd-437f-b428-711f985d0eca\" (UID: \"6f9aa54f-d8cd-437f-b428-711f985d0eca\") " Nov 03 02:51:09.689441 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:09.689263 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f9aa54f-d8cd-437f-b428-711f985d0eca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f9aa54f-d8cd-437f-b428-711f985d0eca-metrics-certs\") pod \"6f9aa54f-d8cd-437f-b428-711f985d0eca\" (UID: \"6f9aa54f-d8cd-437f-b428-711f985d0eca\") " Nov 03 02:51:09.689441 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:09.689354 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f9aa54f-d8cd-437f-b428-711f985d0eca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9aa54f-d8cd-437f-b428-711f985d0eca-openshift-service-ca\") pod \"6f9aa54f-d8cd-437f-b428-711f985d0eca\" (UID: \"6f9aa54f-d8cd-437f-b428-711f985d0eca\") " Nov 03 02:51:09.689585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:09.689455 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f9aa54f-d8cd-437f-b428-711f985d0eca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f9aa54f-d8cd-437f-b428-711f985d0eca-maas-api-tls\") pod \"6f9aa54f-d8cd-437f-b428-711f985d0eca\" (UID: \"6f9aa54f-d8cd-437f-b428-711f985d0eca\") " Nov 03 02:51:09.690383 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:09.690347 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f9aa54f-d8cd-437f-b428-711f985d0eca-openshift-service-ca" pod "6f9aa54f-d8cd-437f-b428-711f985d0eca" (UID: "6f9aa54f-d8cd-437f-b428-711f985d0eca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:51:09.691880 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:09.691853 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9aa54f-d8cd-437f-b428-711f985d0eca-kube-api-access-ls56q" pod "6f9aa54f-d8cd-437f-b428-711f985d0eca" (UID: "6f9aa54f-d8cd-437f-b428-711f985d0eca"). InnerVolumeSpecName "kube-api-access-ls56q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:51:09.691987 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:09.691949 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f9aa54f-d8cd-437f-b428-711f985d0eca-maas-api-tls" pod "6f9aa54f-d8cd-437f-b428-711f985d0eca" (UID: "6f9aa54f-d8cd-437f-b428-711f985d0eca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:51:09.692057 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:09.692000 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f9aa54f-d8cd-437f-b428-711f985d0eca-metrics-certs" pod "6f9aa54f-d8cd-437f-b428-711f985d0eca" (UID: "6f9aa54f-d8cd-437f-b428-711f985d0eca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:51:09.791259 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:09.791145 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f9aa54f-d8cd-437f-b428-711f985d0eca-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:09.791259 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:09.791174 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls56q\" (UniqueName: \"kubernetes.io/projected/6f9aa54f-d8cd-437f-b428-711f985d0eca-kube-api-access-ls56q\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:09.791259 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:09.791186 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f9aa54f-d8cd-437f-b428-711f985d0eca-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:09.791259 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:09.791217 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9aa54f-d8cd-437f-b428-711f985d0eca-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:10.348490 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:10.348465 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" Nov 03 02:51:10.348986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:10.348461 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5" event={"ID":"6f9aa54f-d8cd-437f-b428-711f985d0eca","Type":"ContainerDied","Data":"02503a2ac19617ec71ecfe79cb3e3c9284848d86daec76868e4750e662915f39"} Nov 03 02:51:10.348986 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:10.348596 5463 scope.go:122] "RemoveContainer" containerID="e949b231ac5d0722f66438487002adc2e8f6f1cf244fd238ba1fae98035f71a2" Nov 03 02:51:10.517999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:10.517966 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5"] Nov 03 02:51:10.568014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:10.567988 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e3cc05-699fb46c98-svwp5"] Nov 03 02:51:11.029935 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:11.029886 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:12.443273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:12.443230 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f9aa54f-d8cd-437f-b428-711f985d0eca" path="/var/lib/kubelet/pods/6f9aa54f-d8cd-437f-b428-711f985d0eca/volumes" Nov 03 02:51:13.008650 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:51:13.008616 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:51:15.824345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:15.824314 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:17.025120 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:17.025087 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:17.205082 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:17.205041 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:18.636554 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:18.636507 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:19.166311 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:19.166276 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:21.070762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:21.070722 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:22.422762 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:22.422720 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:22.589918 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:22.589878 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:22.750148 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:22.750057 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:26.051698 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:26.051662 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:26.225208 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:26.225170 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:30.839651 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:30.839614 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:35.659063 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:35.659010 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:36.463915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.463880 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn"] Nov 03 02:51:36.464498 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464485 5463 cpu_manager.go:414] "containerMap: removing container" podUID="66888dec-a1ab-4541-9c61-58014ece2263" containerName="maas-api" Nov 03 02:51:36.464554 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464499 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66888dec-a1ab-4541-9c61-58014ece2263" containerName="maas-api" Nov 03 02:51:36.464554 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464518 5463 cpu_manager.go:414] "containerMap: removing container" podUID="6f9aa54f-d8cd-437f-b428-711f985d0eca" containerName="maas-api" Nov 03 02:51:36.464554 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464524 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f9aa54f-d8cd-437f-b428-711f985d0eca" containerName="maas-api" Nov 03 02:51:36.464554 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464549 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2d2157b2-209f-4ed7-9aab-a79fbe023d82" containerName="istio-proxy" Nov 03 02:51:36.464683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464554 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2157b2-209f-4ed7-9aab-a79fbe023d82" containerName="istio-proxy" Nov 03 02:51:36.464683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464570 5463 cpu_manager.go:414] "containerMap: removing container" podUID="da50d3ef-a1d7-4546-8e52-60ee075543c3" containerName="istio-proxy" Nov 03 02:51:36.464683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464575 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da50d3ef-a1d7-4546-8e52-60ee075543c3" containerName="istio-proxy" Nov 03 02:51:36.464683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464591 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2d2157b2-209f-4ed7-9aab-a79fbe023d82" containerName="istio-proxy" Nov 03 02:51:36.464683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464596 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2157b2-209f-4ed7-9aab-a79fbe023d82" containerName="istio-proxy" Nov 03 02:51:36.464683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464603 5463 cpu_manager.go:414] "containerMap: removing container" podUID="5ec0819e-4232-43a9-820e-3a7b65b7ac9d" containerName="payload-processing" Nov 03 02:51:36.464683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464607 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec0819e-4232-43a9-820e-3a7b65b7ac9d" containerName="payload-processing" Nov 03 02:51:36.464683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464617 5463 cpu_manager.go:414] "containerMap: removing container" podUID="64bcbb35-8a22-43a4-beae-fbd0420e5bdd" containerName="payload-pre-processing" Nov 03 02:51:36.464683 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464622 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64bcbb35-8a22-43a4-beae-fbd0420e5bdd" containerName="payload-pre-processing" Nov 03 02:51:36.464970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464716 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="66888dec-a1ab-4541-9c61-58014ece2263" containerName="maas-api" Nov 03 02:51:36.464970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464725 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f9aa54f-d8cd-437f-b428-711f985d0eca" containerName="maas-api" Nov 03 02:51:36.464970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464736 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2157b2-209f-4ed7-9aab-a79fbe023d82" containerName="istio-proxy" Nov 03 02:51:36.464970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464743 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="da50d3ef-a1d7-4546-8e52-60ee075543c3" containerName="istio-proxy" Nov 03 02:51:36.464970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464752 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2157b2-209f-4ed7-9aab-a79fbe023d82" containerName="istio-proxy" Nov 03 02:51:36.464970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464756 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec0819e-4232-43a9-820e-3a7b65b7ac9d" containerName="payload-processing" Nov 03 02:51:36.464970 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.464763 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="64bcbb35-8a22-43a4-beae-fbd0420e5bdd" containerName="payload-pre-processing" Nov 03 02:51:36.468098 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.468063 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.549060 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.549023 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.549289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.549098 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.549289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.549141 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8q2b\" (UniqueName: \"kubernetes.io/projected/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-kube-api-access-c8q2b\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.549289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.549210 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.549289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.549260 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.549528 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.549329 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.549528 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.549370 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.549528 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.549394 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.549528 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.549462 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.593656 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.593621 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fwpk8\"" Nov 03 02:51:36.598959 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.598923 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn"] Nov 03 02:51:36.650907 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.650871 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.650907 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.650907 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.651152 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.650976 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.651152 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.651103 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.651327 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.651169 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.651327 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.651229 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8q2b\" (UniqueName: \"kubernetes.io/projected/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-kube-api-access-c8q2b\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.651327 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.651289 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.651502 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.651342 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.651502 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.651414 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.652086 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.651781 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.652086 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.651971 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.652086 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.652040 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.652366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.652110 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.652366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.652280 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.654226 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.654161 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.655854 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.655829 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.710614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.710584 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.710934 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.710915 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8q2b\" (UniqueName: \"kubernetes.io/projected/77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e-kube-api-access-c8q2b\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9bjwn\" (UID: \"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:36.782417 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:36.782333 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:37.216946 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:51:37.216915 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77eb3f8e_d87c_4b1c_8e8f_1b9ceff1168e.slice/crio-5c9e95ba5e9e5f08e829e88cd08dca0930b5b4c6fb4d26950c2ea9513d649e6c WatchSource:0}: Error finding container 5c9e95ba5e9e5f08e829e88cd08dca0930b5b4c6fb4d26950c2ea9513d649e6c: Status 404 returned error can't find the container with id 5c9e95ba5e9e5f08e829e88cd08dca0930b5b4c6fb4d26950c2ea9513d649e6c Nov 03 02:51:37.218428 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:37.218404 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn"] Nov 03 02:51:37.219952 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:37.219917 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:51:37.220050 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:37.219982 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:51:37.220050 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:37.220011 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:51:37.561208 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:37.561089 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" event={"ID":"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e","Type":"ContainerStarted","Data":"2b9f069f87551d5624a3aa80110a02531959f299deec7d3ef9acb60d1e169a43"} Nov 03 02:51:37.561208 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:37.561157 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" event={"ID":"77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e","Type":"ContainerStarted","Data":"5c9e95ba5e9e5f08e829e88cd08dca0930b5b4c6fb4d26950c2ea9513d649e6c"} Nov 03 02:51:37.685847 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:37.685793 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" podStartSLOduration=1.6857778350000001 podStartE2EDuration="1.685777835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:51:37.684439168 +0000 UTC m=+3579.368548322" watchObservedRunningTime="2026-11-03 02:51:37.685777835 +0000 UTC m=+3579.369886989" Nov 03 02:51:37.782968 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:37.782929 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:38.785351 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:38.785314 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9bjwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Nov 03 02:51:38.785761 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:38.785366 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" podUID="77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Nov 03 02:51:39.783596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:39.783556 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9bjwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Nov 03 02:51:39.783789 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:39.783619 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" podUID="77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Nov 03 02:51:40.604655 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:40.604609 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:40.783308 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:40.783274 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9bjwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Nov 03 02:51:40.783496 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:40.783335 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" podUID="77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Nov 03 02:51:41.783757 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:41.783724 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9bjwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Nov 03 02:51:41.784247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:41.784220 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" podUID="77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Nov 03 02:51:42.784126 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:42.784089 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9bjwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Nov 03 02:51:42.784545 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:42.784138 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" podUID="77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Nov 03 02:51:43.010616 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:51:43.010583 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:51:43.783108 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:43.783072 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9bjwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Nov 03 02:51:43.783307 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:43.783135 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" podUID="77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Nov 03 02:51:44.782749 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:44.782704 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9bjwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Nov 03 02:51:44.783169 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:44.782780 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" podUID="77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Nov 03 02:51:45.473815 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:45.473769 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:45.653103 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:45.653069 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:45.820688 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:45.820580 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:46.326304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:46.326272 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:46.326900 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:46.326858 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:46.328708 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:46.328663 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" Nov 03 02:51:50.532909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:50.532871 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:55.254620 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:51:55.254581 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:00.057805 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:00.057748 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:00.350717 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:00.350620 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:00.521466 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:00.521430 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:03.074360 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:03.074331 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:52:03.076363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:03.076341 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:52:03.481626 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:03.481592 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:03.680579 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:03.678544 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:04.875714 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:04.875636 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:09.735076 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:09.734990 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:09.898756 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:09.898696 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:10.070619 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:10.070528 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:13.007846 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:52:13.007809 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:52:13.891956 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:13.891923 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:14.621443 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:14.621408 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:15.039112 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:15.039079 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:18.018659 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:18.018626 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s"] Nov 03 02:52:18.019142 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:18.018843 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" podUID="e3926283-2107-4c88-8c94-744886250f75" containerName="istio-proxy" containerID="cri-o://2cb1b26a5c64df734385ed39b1983fc702543dfb919c676a7c9d55fc5d468537" gracePeriod=30 Nov 03 02:52:19.132849 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:19.132812 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:19.320626 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:19.320598 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:19.484802 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:19.484766 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:19.871530 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:19.871452 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:22.422991 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:22.422955 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:22.581578 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:22.581538 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:22.750416 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:22.750340 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc"] Nov 03 02:52:22.756163 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:22.756143 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc" Nov 03 02:52:22.807324 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:22.807296 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc"] Nov 03 02:52:22.870220 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:22.864441 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-95152a-dockercfg-vcvdx\"" Nov 03 02:52:22.870220 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:22.865107 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-95152a\"" Nov 03 02:52:22.918652 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:22.916646 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859202a1-dd87-427e-8e59-acfe73d4b6d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc\" (UID: \"859202a1-dd87-427e-8e59-acfe73d4b6d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc" Nov 03 02:52:22.918652 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:22.918272 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhjq9\" (UniqueName: \"kubernetes.io/projected/859202a1-dd87-427e-8e59-acfe73d4b6d8-kube-api-access-vhjq9\") pod \"payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc\" (UID: \"859202a1-dd87-427e-8e59-acfe73d4b6d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc" Nov 03 02:52:23.020218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.020112 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859202a1-dd87-427e-8e59-acfe73d4b6d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc\" (UID: \"859202a1-dd87-427e-8e59-acfe73d4b6d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc" Nov 03 02:52:23.020218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.020164 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhjq9\" (UniqueName: \"kubernetes.io/projected/859202a1-dd87-427e-8e59-acfe73d4b6d8-kube-api-access-vhjq9\") pod \"payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc\" (UID: \"859202a1-dd87-427e-8e59-acfe73d4b6d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc" Nov 03 02:52:23.020862 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.020840 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859202a1-dd87-427e-8e59-acfe73d4b6d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc\" (UID: \"859202a1-dd87-427e-8e59-acfe73d4b6d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc" Nov 03 02:52:23.075428 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.075398 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:23.078373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.078346 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhjq9\" (UniqueName: \"kubernetes.io/projected/859202a1-dd87-427e-8e59-acfe73d4b6d8-kube-api-access-vhjq9\") pod \"payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc\" (UID: \"859202a1-dd87-427e-8e59-acfe73d4b6d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc" Nov 03 02:52:23.296759 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.296732 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:52:23.323933 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.323909 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-istio-envoy\") pod \"e3926283-2107-4c88-8c94-744886250f75\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " Nov 03 02:52:23.324075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.323947 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3926283-2107-4c88-8c94-744886250f75-kube-api-access-whkdr\" (UniqueName: \"kubernetes.io/projected/e3926283-2107-4c88-8c94-744886250f75-kube-api-access-whkdr\") pod \"e3926283-2107-4c88-8c94-744886250f75\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " Nov 03 02:52:23.324075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.323966 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3926283-2107-4c88-8c94-744886250f75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3926283-2107-4c88-8c94-744886250f75-istio-podinfo\") pod \"e3926283-2107-4c88-8c94-744886250f75\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " Nov 03 02:52:23.324075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.324011 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-workload-certs\") pod \"e3926283-2107-4c88-8c94-744886250f75\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " Nov 03 02:52:23.324075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.324060 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-istio-data\") pod \"e3926283-2107-4c88-8c94-744886250f75\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " Nov 03 02:52:23.324345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.324130 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3926283-2107-4c88-8c94-744886250f75-istio-token\" (UniqueName: \"kubernetes.io/projected/e3926283-2107-4c88-8c94-744886250f75-istio-token\") pod \"e3926283-2107-4c88-8c94-744886250f75\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " Nov 03 02:52:23.324345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.324183 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-credential-socket\") pod \"e3926283-2107-4c88-8c94-744886250f75\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " Nov 03 02:52:23.324345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.324253 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-workload-socket\") pod \"e3926283-2107-4c88-8c94-744886250f75\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " Nov 03 02:52:23.324527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.324348 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-workload-certs" pod "e3926283-2107-4c88-8c94-744886250f75" (UID: "e3926283-2107-4c88-8c94-744886250f75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:52:23.324527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.324364 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3926283-2107-4c88-8c94-744886250f75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3926283-2107-4c88-8c94-744886250f75-istiod-ca-cert\") pod \"e3926283-2107-4c88-8c94-744886250f75\" (UID: \"e3926283-2107-4c88-8c94-744886250f75\") " Nov 03 02:52:23.324527 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.324386 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-istio-data" pod "e3926283-2107-4c88-8c94-744886250f75" (UID: "e3926283-2107-4c88-8c94-744886250f75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:52:23.324701 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.324547 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-workload-socket" pod "e3926283-2107-4c88-8c94-744886250f75" (UID: "e3926283-2107-4c88-8c94-744886250f75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:52:23.324701 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.324631 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-credential-socket" pod "e3926283-2107-4c88-8c94-744886250f75" (UID: "e3926283-2107-4c88-8c94-744886250f75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:52:23.324821 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.324792 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3926283-2107-4c88-8c94-744886250f75-istiod-ca-cert" pod "e3926283-2107-4c88-8c94-744886250f75" (UID: "e3926283-2107-4c88-8c94-744886250f75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:52:23.325104 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.325081 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3926283-2107-4c88-8c94-744886250f75-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:23.325184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.325109 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:23.325184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.325127 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:23.325184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.325141 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:23.325184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.325159 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:23.326408 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.326367 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3926283-2107-4c88-8c94-744886250f75-kube-api-access-whkdr" pod "e3926283-2107-4c88-8c94-744886250f75" (UID: "e3926283-2107-4c88-8c94-744886250f75"). InnerVolumeSpecName "kube-api-access-whkdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:52:23.326408 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.326371 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3926283-2107-4c88-8c94-744886250f75-istio-podinfo" pod "e3926283-2107-4c88-8c94-744886250f75" (UID: "e3926283-2107-4c88-8c94-744886250f75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:52:23.326408 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.326376 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-istio-envoy" pod "e3926283-2107-4c88-8c94-744886250f75" (UID: "e3926283-2107-4c88-8c94-744886250f75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:52:23.326650 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.326520 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3926283-2107-4c88-8c94-744886250f75-istio-token" pod "e3926283-2107-4c88-8c94-744886250f75" (UID: "e3926283-2107-4c88-8c94-744886250f75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:52:23.371699 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.371672 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc" Nov 03 02:52:23.426508 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.426400 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3926283-2107-4c88-8c94-744886250f75-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:23.426508 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.426447 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whkdr\" (UniqueName: \"kubernetes.io/projected/e3926283-2107-4c88-8c94-744886250f75-kube-api-access-whkdr\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:23.426508 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.426465 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3926283-2107-4c88-8c94-744886250f75-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:23.426508 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.426482 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3926283-2107-4c88-8c94-744886250f75-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:23.746874 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:52:23.746847 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859202a1_dd87_427e_8e59_acfe73d4b6d8.slice/crio-fc9cee867ea41130db12f7aa6cb78c651645da890375e5b3fb0f3f5aa1246f7f WatchSource:0}: Error finding container fc9cee867ea41130db12f7aa6cb78c651645da890375e5b3fb0f3f5aa1246f7f: Status 404 returned error can't find the container with id fc9cee867ea41130db12f7aa6cb78c651645da890375e5b3fb0f3f5aa1246f7f Nov 03 02:52:23.749694 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.749670 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc"] Nov 03 02:52:23.956429 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.956392 5463 generic.go:356] "Generic (PLEG): container finished" podID="e3926283-2107-4c88-8c94-744886250f75" containerID="2cb1b26a5c64df734385ed39b1983fc702543dfb919c676a7c9d55fc5d468537" exitCode=0 Nov 03 02:52:23.956623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.956435 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" event={"ID":"e3926283-2107-4c88-8c94-744886250f75","Type":"ContainerDied","Data":"2cb1b26a5c64df734385ed39b1983fc702543dfb919c676a7c9d55fc5d468537"} Nov 03 02:52:23.956623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.956465 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" Nov 03 02:52:23.956623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.956530 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s" event={"ID":"e3926283-2107-4c88-8c94-744886250f75","Type":"ContainerDied","Data":"a1b7843fe90a95cc01f0aad60eebc280f91499c44843ccb6d7a4ae82e1de464a"} Nov 03 02:52:23.956623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.956560 5463 scope.go:122] "RemoveContainer" containerID="2cb1b26a5c64df734385ed39b1983fc702543dfb919c676a7c9d55fc5d468537" Nov 03 02:52:23.958650 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.958616 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc" event={"ID":"859202a1-dd87-427e-8e59-acfe73d4b6d8","Type":"ContainerStarted","Data":"8d52c31e4d9d3bdbc4a134742d43935b77446382e1e3f2036c3d07972032ad77"} Nov 03 02:52:23.958791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.958677 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc" event={"ID":"859202a1-dd87-427e-8e59-acfe73d4b6d8","Type":"ContainerStarted","Data":"fc9cee867ea41130db12f7aa6cb78c651645da890375e5b3fb0f3f5aa1246f7f"} Nov 03 02:52:23.958791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.958710 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc" Nov 03 02:52:23.968074 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.968056 5463 scope.go:122] "RemoveContainer" containerID="2cb1b26a5c64df734385ed39b1983fc702543dfb919c676a7c9d55fc5d468537" Nov 03 02:52:23.968406 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:52:23.968385 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb1b26a5c64df734385ed39b1983fc702543dfb919c676a7c9d55fc5d468537\": container with ID starting with 2cb1b26a5c64df734385ed39b1983fc702543dfb919c676a7c9d55fc5d468537 not found: ID does not exist" containerID="2cb1b26a5c64df734385ed39b1983fc702543dfb919c676a7c9d55fc5d468537" Nov 03 02:52:23.968489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:23.968415 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb1b26a5c64df734385ed39b1983fc702543dfb919c676a7c9d55fc5d468537"} err="failed to get container status \"2cb1b26a5c64df734385ed39b1983fc702543dfb919c676a7c9d55fc5d468537\": rpc error: code = NotFound desc = could not find container \"2cb1b26a5c64df734385ed39b1983fc702543dfb919c676a7c9d55fc5d468537\": container with ID starting with 2cb1b26a5c64df734385ed39b1983fc702543dfb919c676a7c9d55fc5d468537 not found: ID does not exist" Nov 03 02:52:24.126790 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.126694 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc" podStartSLOduration=2.126677256 podStartE2EDuration="2.126677256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:52:24.123663407 +0000 UTC m=+3625.807772564" watchObservedRunningTime="2026-11-03 02:52:24.126677256 +0000 UTC m=+3625.810786451" Nov 03 02:52:24.238886 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.238843 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s"] Nov 03 02:52:24.243851 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.243822 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-njk9s"] Nov 03 02:52:24.414811 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.414778 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994"] Nov 03 02:52:24.415006 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.414981 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" podUID="744caa65-978b-4a72-af47-fc8db4ad7f02" containerName="maas-api" containerID="cri-o://72bee3e9e44fd0a2db73451009282ff8026f6c3904677e657f7e51a79a435c01" gracePeriod=30 Nov 03 02:52:24.448689 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.448658 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3926283-2107-4c88-8c94-744886250f75" path="/var/lib/kubelet/pods/e3926283-2107-4c88-8c94-744886250f75/volumes" Nov 03 02:52:24.532651 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.532514 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:24.703999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.703972 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" Nov 03 02:52:24.744725 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.744697 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/744caa65-978b-4a72-af47-fc8db4ad7f02-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/744caa65-978b-4a72-af47-fc8db4ad7f02-openshift-service-ca\") pod \"744caa65-978b-4a72-af47-fc8db4ad7f02\" (UID: \"744caa65-978b-4a72-af47-fc8db4ad7f02\") " Nov 03 02:52:24.744874 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.744740 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/744caa65-978b-4a72-af47-fc8db4ad7f02-metrics-certs\" (UniqueName: \"kubernetes.io/secret/744caa65-978b-4a72-af47-fc8db4ad7f02-metrics-certs\") pod \"744caa65-978b-4a72-af47-fc8db4ad7f02\" (UID: \"744caa65-978b-4a72-af47-fc8db4ad7f02\") " Nov 03 02:52:24.744874 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.744780 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/744caa65-978b-4a72-af47-fc8db4ad7f02-kube-api-access-w2lks\" (UniqueName: \"kubernetes.io/projected/744caa65-978b-4a72-af47-fc8db4ad7f02-kube-api-access-w2lks\") pod \"744caa65-978b-4a72-af47-fc8db4ad7f02\" (UID: \"744caa65-978b-4a72-af47-fc8db4ad7f02\") " Nov 03 02:52:24.744990 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.744973 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/744caa65-978b-4a72-af47-fc8db4ad7f02-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/744caa65-978b-4a72-af47-fc8db4ad7f02-maas-api-tls\") pod \"744caa65-978b-4a72-af47-fc8db4ad7f02\" (UID: \"744caa65-978b-4a72-af47-fc8db4ad7f02\") " Nov 03 02:52:24.745053 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.745015 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744caa65-978b-4a72-af47-fc8db4ad7f02-openshift-service-ca" pod "744caa65-978b-4a72-af47-fc8db4ad7f02" (UID: "744caa65-978b-4a72-af47-fc8db4ad7f02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:52:24.745380 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.745361 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/744caa65-978b-4a72-af47-fc8db4ad7f02-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:24.746981 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.746954 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/744caa65-978b-4a72-af47-fc8db4ad7f02-metrics-certs" pod "744caa65-978b-4a72-af47-fc8db4ad7f02" (UID: "744caa65-978b-4a72-af47-fc8db4ad7f02"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:52:24.747075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.747010 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744caa65-978b-4a72-af47-fc8db4ad7f02-kube-api-access-w2lks" pod "744caa65-978b-4a72-af47-fc8db4ad7f02" (UID: "744caa65-978b-4a72-af47-fc8db4ad7f02"). InnerVolumeSpecName "kube-api-access-w2lks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:52:24.747075 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.747016 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/744caa65-978b-4a72-af47-fc8db4ad7f02-maas-api-tls" pod "744caa65-978b-4a72-af47-fc8db4ad7f02" (UID: "744caa65-978b-4a72-af47-fc8db4ad7f02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:52:24.846860 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.846827 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/744caa65-978b-4a72-af47-fc8db4ad7f02-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:24.846860 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.846857 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2lks\" (UniqueName: \"kubernetes.io/projected/744caa65-978b-4a72-af47-fc8db4ad7f02-kube-api-access-w2lks\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:24.846860 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.846867 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/744caa65-978b-4a72-af47-fc8db4ad7f02-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:24.973184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.973095 5463 generic.go:356] "Generic (PLEG): container finished" podID="744caa65-978b-4a72-af47-fc8db4ad7f02" containerID="72bee3e9e44fd0a2db73451009282ff8026f6c3904677e657f7e51a79a435c01" exitCode=0 Nov 03 02:52:24.973184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.973147 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" event={"ID":"744caa65-978b-4a72-af47-fc8db4ad7f02","Type":"ContainerDied","Data":"72bee3e9e44fd0a2db73451009282ff8026f6c3904677e657f7e51a79a435c01"} Nov 03 02:52:24.973467 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.973159 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" Nov 03 02:52:24.973467 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.973257 5463 scope.go:122] "RemoveContainer" containerID="72bee3e9e44fd0a2db73451009282ff8026f6c3904677e657f7e51a79a435c01" Nov 03 02:52:24.973467 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.973243 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994" event={"ID":"744caa65-978b-4a72-af47-fc8db4ad7f02","Type":"ContainerDied","Data":"bc78a2a3184c041c218ebed618a9c9e385f2564def19a9ef7d9e2605a7df7e78"} Nov 03 02:52:24.986590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.986523 5463 scope.go:122] "RemoveContainer" containerID="72bee3e9e44fd0a2db73451009282ff8026f6c3904677e657f7e51a79a435c01" Nov 03 02:52:24.986939 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:52:24.986912 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72bee3e9e44fd0a2db73451009282ff8026f6c3904677e657f7e51a79a435c01\": container with ID starting with 72bee3e9e44fd0a2db73451009282ff8026f6c3904677e657f7e51a79a435c01 not found: ID does not exist" containerID="72bee3e9e44fd0a2db73451009282ff8026f6c3904677e657f7e51a79a435c01" Nov 03 02:52:24.987025 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:24.986943 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72bee3e9e44fd0a2db73451009282ff8026f6c3904677e657f7e51a79a435c01"} err="failed to get container status \"72bee3e9e44fd0a2db73451009282ff8026f6c3904677e657f7e51a79a435c01\": rpc error: code = NotFound desc = could not find container \"72bee3e9e44fd0a2db73451009282ff8026f6c3904677e657f7e51a79a435c01\": container with ID starting with 72bee3e9e44fd0a2db73451009282ff8026f6c3904677e657f7e51a79a435c01 not found: ID does not exist" Nov 03 02:52:25.141229 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:25.140610 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994"] Nov 03 02:52:25.143584 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:25.143555 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c8ee6b-6b8b548978-qt994"] Nov 03 02:52:26.216128 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:26.216089 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:26.394412 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:26.394370 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:26.444740 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:26.444700 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="744caa65-978b-4a72-af47-fc8db4ad7f02" path="/var/lib/kubelet/pods/744caa65-978b-4a72-af47-fc8db4ad7f02/volumes" Nov 03 02:52:26.748565 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:26.748531 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:27.925215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:27.924847 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:28.727352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.727319 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh"] Nov 03 02:52:28.727997 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.727978 5463 cpu_manager.go:414] "containerMap: removing container" podUID="744caa65-978b-4a72-af47-fc8db4ad7f02" containerName="maas-api" Nov 03 02:52:28.727997 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.727995 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="744caa65-978b-4a72-af47-fc8db4ad7f02" containerName="maas-api" Nov 03 02:52:28.728211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.728015 5463 cpu_manager.go:414] "containerMap: removing container" podUID="e3926283-2107-4c88-8c94-744886250f75" containerName="istio-proxy" Nov 03 02:52:28.728211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.728020 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3926283-2107-4c88-8c94-744886250f75" containerName="istio-proxy" Nov 03 02:52:28.728211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.728108 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="744caa65-978b-4a72-af47-fc8db4ad7f02" containerName="maas-api" Nov 03 02:52:28.728211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.728119 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3926283-2107-4c88-8c94-744886250f75" containerName="istio-proxy" Nov 03 02:52:28.733141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.733118 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.805422 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.805391 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-envoy\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.805594 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.805471 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-token\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.805663 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.805593 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spc47\" (UniqueName: \"kubernetes.io/projected/bc8f16d1-4d9c-4c4b-8449-689172ef80be-kube-api-access-spc47\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.805663 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.805635 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-data\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.805786 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.805755 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-credential-socket\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.805856 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.805839 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-workload-socket\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.805900 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.805885 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-podinfo\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.805943 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.805930 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-workload-certs\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.806008 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.805993 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istiod-ca-cert\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.833804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.833763 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh"] Nov 03 02:52:28.840235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.836924 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:28.842687 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.842658 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ecdbc7-openshift-default-dockercfg-qw6s5\"" Nov 03 02:52:28.907669 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.907634 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-envoy\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.908924 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.908897 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-token\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.909173 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.909156 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spc47\" (UniqueName: \"kubernetes.io/projected/bc8f16d1-4d9c-4c4b-8449-689172ef80be-kube-api-access-spc47\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.909306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.909292 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-data\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.909443 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.909431 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-credential-socket\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.909570 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.909558 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-workload-socket\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.909686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.909674 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-podinfo\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.909794 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.909782 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-workload-certs\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.909917 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.909905 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istiod-ca-cert\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.914469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.910748 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istiod-ca-cert\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.918366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.911274 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-envoy\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.918366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.911529 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-credential-socket\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.918366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.912295 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-data\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.918366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.915277 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-podinfo\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.918366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.915541 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-workload-socket\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.918366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.915823 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-workload-certs\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.982870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.982780 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-token\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:28.983357 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:28.982995 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spc47\" (UniqueName: \"kubernetes.io/projected/bc8f16d1-4d9c-4c4b-8449-689172ef80be-kube-api-access-spc47\") pod \"e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:29.049827 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:29.049781 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:29.239729 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:29.239635 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:29.512834 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:29.512791 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh"] Nov 03 02:52:29.514949 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:29.514918 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:52:29.515061 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:29.514996 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:52:29.515061 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:29.515042 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:52:30.021414 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:30.021380 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" event={"ID":"bc8f16d1-4d9c-4c4b-8449-689172ef80be","Type":"ContainerStarted","Data":"12cf0db4a1a4f1e00c579a1aaa3e4b354df4638f8b28dd474a8915bdbf9fbff9"} Nov 03 02:52:30.021946 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:30.021434 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" event={"ID":"bc8f16d1-4d9c-4c4b-8449-689172ef80be","Type":"ContainerStarted","Data":"463c6a3b12058a121d266ba2f018e30748b4d3ad867227ddb3ed1e8877f51a95"} Nov 03 02:52:30.050977 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:30.050942 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:30.052995 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:30.052960 5463 patch_prober.go:29] interesting pod/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Nov 03 02:52:30.053117 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:30.053013 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" podUID="bc8f16d1-4d9c-4c4b-8449-689172ef80be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Nov 03 02:52:30.153888 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:30.153821 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" podStartSLOduration=2.153800237 podStartE2EDuration="2.153800237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:52:30.148062439 +0000 UTC m=+3631.832171624" watchObservedRunningTime="2026-11-03 02:52:30.153800237 +0000 UTC m=+3631.837909417" Nov 03 02:52:31.080213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:31.079309 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:31.481157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:31.481123 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:32.086186 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:32.086153 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:32.087288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:32.087262 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:52:33.195432 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:33.195400 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:33.381374 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:33.381324 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:33.556327 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:33.556233 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:34.789368 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:34.789333 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:34.946135 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:34.946098 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:34.976927 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:34.976900 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc" Nov 03 02:52:36.441096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:36.441061 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:36.601797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:36.601761 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:36.763479 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:36.763391 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:36.834235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:36.834186 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9"] Nov 03 02:52:36.838447 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:36.838419 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9" Nov 03 02:52:36.891253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:36.891181 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9"] Nov 03 02:52:36.947218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:36.943430 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ecdbc7\"" Nov 03 02:52:36.969322 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:36.969279 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a032506e-d4ff-49ea-a5e2-a1fa2137b3d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9\" (UID: \"a032506e-d4ff-49ea-a5e2-a1fa2137b3d4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9" Nov 03 02:52:36.972383 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:36.972351 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn7vd\" (UniqueName: \"kubernetes.io/projected/a032506e-d4ff-49ea-a5e2-a1fa2137b3d4-kube-api-access-gn7vd\") pod \"payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9\" (UID: \"a032506e-d4ff-49ea-a5e2-a1fa2137b3d4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9" Nov 03 02:52:36.972717 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:36.972695 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ecdbc7-dockercfg-xzt25\"" Nov 03 02:52:37.073534 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:37.073452 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a032506e-d4ff-49ea-a5e2-a1fa2137b3d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9\" (UID: \"a032506e-d4ff-49ea-a5e2-a1fa2137b3d4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9" Nov 03 02:52:37.073534 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:37.073517 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn7vd\" (UniqueName: \"kubernetes.io/projected/a032506e-d4ff-49ea-a5e2-a1fa2137b3d4-kube-api-access-gn7vd\") pod \"payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9\" (UID: \"a032506e-d4ff-49ea-a5e2-a1fa2137b3d4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9" Nov 03 02:52:37.074274 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:37.074248 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a032506e-d4ff-49ea-a5e2-a1fa2137b3d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9\" (UID: \"a032506e-d4ff-49ea-a5e2-a1fa2137b3d4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9" Nov 03 02:52:37.132873 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:37.132840 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn7vd\" (UniqueName: \"kubernetes.io/projected/a032506e-d4ff-49ea-a5e2-a1fa2137b3d4-kube-api-access-gn7vd\") pod \"payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9\" (UID: \"a032506e-d4ff-49ea-a5e2-a1fa2137b3d4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9" Nov 03 02:52:37.153515 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:37.153482 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9" Nov 03 02:52:37.525000 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:52:37.524969 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda032506e_d4ff_49ea_a5e2_a1fa2137b3d4.slice/crio-effc21cdbd48b3e8d55cfd2663f72cee019ca3460b97b876ffc67a239ba5f05d WatchSource:0}: Error finding container effc21cdbd48b3e8d55cfd2663f72cee019ca3460b97b876ffc67a239ba5f05d: Status 404 returned error can't find the container with id effc21cdbd48b3e8d55cfd2663f72cee019ca3460b97b876ffc67a239ba5f05d Nov 03 02:52:37.528187 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:37.528158 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9"] Nov 03 02:52:38.131463 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:38.131429 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9" event={"ID":"a032506e-d4ff-49ea-a5e2-a1fa2137b3d4","Type":"ContainerStarted","Data":"0792f9df29ecc4a52e4e5ef1d9e5b8b6695647a8ae2b4a2afa61791ac3f3ae7a"} Nov 03 02:52:38.131658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:38.131483 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9" event={"ID":"a032506e-d4ff-49ea-a5e2-a1fa2137b3d4","Type":"ContainerStarted","Data":"effc21cdbd48b3e8d55cfd2663f72cee019ca3460b97b876ffc67a239ba5f05d"} Nov 03 02:52:38.131658 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:38.131517 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9" Nov 03 02:52:38.253523 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:38.253472 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9" podStartSLOduration=2.253459216 podStartE2EDuration="2.253459216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:52:38.250915629 +0000 UTC m=+3639.935024788" watchObservedRunningTime="2026-11-03 02:52:38.253459216 +0000 UTC m=+3639.937568415" Nov 03 02:52:41.305181 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:41.305141 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:43.009943 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:52:43.009908 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:52:46.055519 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:46.055466 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:49.141559 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:49.141528 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9" Nov 03 02:52:50.949416 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:50.949351 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:51.960230 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:51.960185 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb"] Nov 03 02:52:51.964962 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:51.964939 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.057581 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.057539 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.057773 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.057597 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.057853 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.057815 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.057912 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.057873 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.057965 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.057940 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.058060 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.058043 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.058123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.058076 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv4fr\" (UniqueName: \"kubernetes.io/projected/88e27111-aadb-40b3-a564-cf0b3a6f86bb-kube-api-access-hv4fr\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.058179 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.058142 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.058248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.058227 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.071543 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.071516 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb"] Nov 03 02:52:52.159573 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.159532 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.159573 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.159577 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.159828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.159724 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.159828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.159767 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.159828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.159793 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.160015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.159842 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.160015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.159869 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv4fr\" (UniqueName: \"kubernetes.io/projected/88e27111-aadb-40b3-a564-cf0b3a6f86bb-kube-api-access-hv4fr\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.160015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.159906 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.160015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.159931 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.160015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.159942 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.160015 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.159966 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.160334 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.160083 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.160334 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.160171 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.160632 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.160614 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.162265 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.162232 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.162511 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.162493 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.218759 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.218698 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv4fr\" (UniqueName: \"kubernetes.io/projected/88e27111-aadb-40b3-a564-cf0b3a6f86bb-kube-api-access-hv4fr\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.219145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.219121 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-97lfb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.279663 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.279632 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:52.703417 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:52:52.703388 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e27111_aadb_40b3_a564_cf0b3a6f86bb.slice/crio-b648081012724fad1ab63b9b9bdf1630a1090ffd787cd887b32f6774c1f15d10 WatchSource:0}: Error finding container b648081012724fad1ab63b9b9bdf1630a1090ffd787cd887b32f6774c1f15d10: Status 404 returned error can't find the container with id b648081012724fad1ab63b9b9bdf1630a1090ffd787cd887b32f6774c1f15d10 Nov 03 02:52:52.706712 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.706678 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:52:52.706808 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.706745 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb"] Nov 03 02:52:52.706808 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.706748 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:52:52.706887 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:52.706834 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:52:53.245141 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:53.245106 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" event={"ID":"88e27111-aadb-40b3-a564-cf0b3a6f86bb","Type":"ContainerStarted","Data":"3f41768e1a9ece49aea430c3109e6ec3ac881da64df844e2b5a608dd2bace20c"} Nov 03 02:52:53.245578 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:53.245167 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" event={"ID":"88e27111-aadb-40b3-a564-cf0b3a6f86bb","Type":"ContainerStarted","Data":"b648081012724fad1ab63b9b9bdf1630a1090ffd787cd887b32f6774c1f15d10"} Nov 03 02:52:53.279964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:53.279932 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:52:53.281389 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:53.281356 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-97lfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Nov 03 02:52:53.281509 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:53.281410 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" podUID="88e27111-aadb-40b3-a564-cf0b3a6f86bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Nov 03 02:52:53.366829 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:53.366757 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" podStartSLOduration=2.366734912 podStartE2EDuration="2.366734912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:52:53.365213101 +0000 UTC m=+3655.049322260" watchObservedRunningTime="2026-11-03 02:52:53.366734912 +0000 UTC m=+3655.050844068" Nov 03 02:52:54.280939 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:54.280904 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-97lfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Nov 03 02:52:54.281368 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:54.280959 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" podUID="88e27111-aadb-40b3-a564-cf0b3a6f86bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Nov 03 02:52:55.280735 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:55.280702 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-97lfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Nov 03 02:52:55.280939 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:55.280763 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" podUID="88e27111-aadb-40b3-a564-cf0b3a6f86bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Nov 03 02:52:55.767688 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:55.767640 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:56.281049 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:56.281016 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-97lfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Nov 03 02:52:56.281273 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:56.281068 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" podUID="88e27111-aadb-40b3-a564-cf0b3a6f86bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Nov 03 02:52:57.280569 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:57.280535 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-97lfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Nov 03 02:52:57.280994 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:57.280587 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" podUID="88e27111-aadb-40b3-a564-cf0b3a6f86bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Nov 03 02:52:58.280182 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:58.280139 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-97lfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Nov 03 02:52:58.280378 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:58.280235 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" podUID="88e27111-aadb-40b3-a564-cf0b3a6f86bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Nov 03 02:52:59.280112 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:59.280069 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-97lfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Nov 03 02:52:59.280619 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:52:59.280154 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" podUID="88e27111-aadb-40b3-a564-cf0b3a6f86bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Nov 03 02:53:00.280807 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:00.280762 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-97lfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Nov 03 02:53:00.281345 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:00.280824 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" podUID="88e27111-aadb-40b3-a564-cf0b3a6f86bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Nov 03 02:53:00.521013 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:00.520948 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:01.461926 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:01.461895 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:53:01.462692 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:01.462663 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:53:01.463371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:01.463354 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 02:53:05.337070 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:05.337033 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:09.560359 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:09.560323 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:10.432885 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:10.432846 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:10.963052 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:10.962988 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:11.539589 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.539559 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr"] Nov 03 02:53:11.545530 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.545508 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.599595 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.599560 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-dc14f6-openshift-default-dockercfg-m277s\"" Nov 03 02:53:11.644601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.644564 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:11.646393 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.646359 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b38f888-2173-4d9b-a4cb-047d5086e894-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.646551 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.646513 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.646633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.646599 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.646697 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.646674 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.646758 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.646735 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.646822 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.646801 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.647084 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.647027 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.647238 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.647163 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th7kw\" (UniqueName: \"kubernetes.io/projected/3b38f888-2173-4d9b-a4cb-047d5086e894-kube-api-access-th7kw\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.647335 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.647269 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.655284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.655261 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr"] Nov 03 02:53:11.748042 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.748009 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.748261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.748062 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.748261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.748091 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.748261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.748137 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.748261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.748255 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.748501 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.748303 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th7kw\" (UniqueName: \"kubernetes.io/projected/3b38f888-2173-4d9b-a4cb-047d5086e894-kube-api-access-th7kw\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.748501 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.748335 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.748501 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.748408 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b38f888-2173-4d9b-a4cb-047d5086e894-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.748501 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.748464 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.748501 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.748489 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.748816 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.748540 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.748816 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.748803 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.748943 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.748893 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.749143 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.749123 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b38f888-2173-4d9b-a4cb-047d5086e894-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.750396 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.750376 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.750681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.750663 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.813435 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.813352 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th7kw\" (UniqueName: \"kubernetes.io/projected/3b38f888-2173-4d9b-a4cb-047d5086e894-kube-api-access-th7kw\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.817763 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.817732 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:11.837804 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.837777 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:11.874330 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:11.874305 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:12.298336 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:12.298306 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr"] Nov 03 02:53:12.330871 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:53:12.330843 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b38f888_2173_4d9b_a4cb_047d5086e894.slice/crio-4d8561150ed785d50c0ff0fc5df84bde3fc42a76ecbf80c6e572306b5506d283 WatchSource:0}: Error finding container 4d8561150ed785d50c0ff0fc5df84bde3fc42a76ecbf80c6e572306b5506d283: Status 404 returned error can't find the container with id 4d8561150ed785d50c0ff0fc5df84bde3fc42a76ecbf80c6e572306b5506d283 Nov 03 02:53:12.334046 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:12.334010 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:53:12.334156 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:12.334099 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:53:12.334156 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:12.334135 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:53:12.408875 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:12.408850 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" event={"ID":"3b38f888-2173-4d9b-a4cb-047d5086e894","Type":"ContainerStarted","Data":"4d8561150ed785d50c0ff0fc5df84bde3fc42a76ecbf80c6e572306b5506d283"} Nov 03 02:53:12.530322 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:12.530265 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" podStartSLOduration=1.530251958 podStartE2EDuration="1.530251958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:53:12.527384207 +0000 UTC m=+3674.211493362" watchObservedRunningTime="2026-11-03 02:53:12.530251958 +0000 UTC m=+3674.214361118" Nov 03 02:53:12.874611 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:12.874578 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:13.008488 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:53:13.008450 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:53:13.419267 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:13.419223 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" event={"ID":"3b38f888-2173-4d9b-a4cb-047d5086e894","Type":"ContainerStarted","Data":"66e900090ec0db270731d1f9c0a91e33778a6d64c6f796dbb40a6cdb762dce19"} Nov 03 02:53:13.881915 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:13.881828 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:14.427263 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:14.427234 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:14.428242 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:14.428222 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:53:15.437977 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:15.437935 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:15.599413 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:15.599374 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:15.760333 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:15.760238 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:18.707493 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:18.707425 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:18.897668 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:18.897629 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:20.389248 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:20.389184 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:23.402129 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:23.402089 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:23.561431 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:23.561392 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:23.716808 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:23.716721 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:25.111929 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:25.111888 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:29.872123 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:29.872090 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:34.268802 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:34.268770 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt"] Nov 03 02:53:34.272179 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:34.272162 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" Nov 03 02:53:34.326720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:34.326678 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt"] Nov 03 02:53:34.333404 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:34.333379 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/442cb483-89c6-4e1e-b19c-43e2741ee83d-metrics-certs\") pod \"maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt\" (UID: \"442cb483-89c6-4e1e-b19c-43e2741ee83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" Nov 03 02:53:34.333547 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:34.333448 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/442cb483-89c6-4e1e-b19c-43e2741ee83d-maas-api-tls\") pod \"maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt\" (UID: \"442cb483-89c6-4e1e-b19c-43e2741ee83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" Nov 03 02:53:34.333652 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:34.333635 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckblm\" (UniqueName: \"kubernetes.io/projected/442cb483-89c6-4e1e-b19c-43e2741ee83d-kube-api-access-ckblm\") pod \"maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt\" (UID: \"442cb483-89c6-4e1e-b19c-43e2741ee83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" Nov 03 02:53:34.333709 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:34.333676 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/442cb483-89c6-4e1e-b19c-43e2741ee83d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt\" (UID: \"442cb483-89c6-4e1e-b19c-43e2741ee83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" Nov 03 02:53:34.373805 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:34.373776 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-95152a-serving-cert\"" Nov 03 02:53:34.434909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:34.434864 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/442cb483-89c6-4e1e-b19c-43e2741ee83d-maas-api-tls\") pod \"maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt\" (UID: \"442cb483-89c6-4e1e-b19c-43e2741ee83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" Nov 03 02:53:34.435089 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:53:34.435015 5463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-serving-cert: secret "maas-api-e2e-inf-a-95152a-serving-cert" not found Nov 03 02:53:34.435089 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:53:34.435075 5463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/442cb483-89c6-4e1e-b19c-43e2741ee83d-maas-api-tls podName:442cb483-89c6-4e1e-b19c-43e2741ee83d nodeName:}" failed. No retries permitted until 2026-11-03 02:53:34.935055823 +0000 UTC m=+3696.619164973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/442cb483-89c6-4e1e-b19c-43e2741ee83d-maas-api-tls") pod "maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" (UID: "442cb483-89c6-4e1e-b19c-43e2741ee83d") : secret "maas-api-e2e-inf-a-95152a-serving-cert" not found Nov 03 02:53:34.435223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:34.435101 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckblm\" (UniqueName: \"kubernetes.io/projected/442cb483-89c6-4e1e-b19c-43e2741ee83d-kube-api-access-ckblm\") pod \"maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt\" (UID: \"442cb483-89c6-4e1e-b19c-43e2741ee83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" Nov 03 02:53:34.435223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:34.435149 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/442cb483-89c6-4e1e-b19c-43e2741ee83d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt\" (UID: \"442cb483-89c6-4e1e-b19c-43e2741ee83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" Nov 03 02:53:34.435223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:34.435219 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/442cb483-89c6-4e1e-b19c-43e2741ee83d-metrics-certs\") pod \"maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt\" (UID: \"442cb483-89c6-4e1e-b19c-43e2741ee83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" Nov 03 02:53:34.436115 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:34.436086 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/442cb483-89c6-4e1e-b19c-43e2741ee83d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt\" (UID: \"442cb483-89c6-4e1e-b19c-43e2741ee83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" Nov 03 02:53:34.437869 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:34.437849 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/442cb483-89c6-4e1e-b19c-43e2741ee83d-metrics-certs\") pod \"maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt\" (UID: \"442cb483-89c6-4e1e-b19c-43e2741ee83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" Nov 03 02:53:34.481469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:34.481439 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:34.492588 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:34.492563 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckblm\" (UniqueName: \"kubernetes.io/projected/442cb483-89c6-4e1e-b19c-43e2741ee83d-kube-api-access-ckblm\") pod \"maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt\" (UID: \"442cb483-89c6-4e1e-b19c-43e2741ee83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" Nov 03 02:53:34.652413 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:34.652382 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:34.796801 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:34.796771 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:34.941802 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:34.941715 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/442cb483-89c6-4e1e-b19c-43e2741ee83d-maas-api-tls\") pod \"maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt\" (UID: \"442cb483-89c6-4e1e-b19c-43e2741ee83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" Nov 03 02:53:34.944661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:34.944631 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/442cb483-89c6-4e1e-b19c-43e2741ee83d-maas-api-tls\") pod \"maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt\" (UID: \"442cb483-89c6-4e1e-b19c-43e2741ee83d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" Nov 03 02:53:35.189938 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:35.189904 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" Nov 03 02:53:35.561663 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:53:35.561630 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442cb483_89c6_4e1e_b19c_43e2741ee83d.slice/crio-5e3d6edabf1586a8238a33b47006774a91fcaf093e7b0de9e9d49cd31ec50f26 WatchSource:0}: Error finding container 5e3d6edabf1586a8238a33b47006774a91fcaf093e7b0de9e9d49cd31ec50f26: Status 404 returned error can't find the container with id 5e3d6edabf1586a8238a33b47006774a91fcaf093e7b0de9e9d49cd31ec50f26 Nov 03 02:53:35.563257 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:35.563235 5463 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 02:53:35.564825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:35.564801 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt"] Nov 03 02:53:35.611371 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:35.611342 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" event={"ID":"442cb483-89c6-4e1e-b19c-43e2741ee83d","Type":"ContainerStarted","Data":"5e3d6edabf1586a8238a33b47006774a91fcaf093e7b0de9e9d49cd31ec50f26"} Nov 03 02:53:36.619531 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:36.619491 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" event={"ID":"442cb483-89c6-4e1e-b19c-43e2741ee83d","Type":"ContainerStarted","Data":"66d631f01d33b4caa7855a3afd79847ef4bd32aa058a434c463a81bd82d547d8"} Nov 03 02:53:36.619904 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:36.619551 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" Nov 03 02:53:36.761310 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:36.761177 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" podStartSLOduration=1.8460147679999999 podStartE2EDuration="2.761162673s" totalImagesPullingTime="915.147905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:53:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:53:36.760473371 +0000 UTC m=+3698.444582526" watchObservedRunningTime="2026-11-03 02:53:36.761162673 +0000 UTC m=+3698.445271828" Nov 03 02:53:39.433924 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:39.433840 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:42.633055 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:42.633022 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" Nov 03 02:53:43.008680 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:53:43.008649 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:53:44.233414 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:44.233382 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:48.890897 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:48.890709 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:49.056822 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:49.056786 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:49.221532 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:49.221445 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:52.379585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.379554 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt"] Nov 03 02:53:52.382776 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.382756 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.469220 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.469164 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.469391 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.469367 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.469462 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.469444 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.469584 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.469559 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.469657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.469604 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.469737 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.469711 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.469796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.469773 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.469867 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.469849 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v58dh\" (UniqueName: \"kubernetes.io/projected/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-kube-api-access-v58dh\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.469931 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.469891 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.478662 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.478634 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt"] Nov 03 02:53:52.571302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.571268 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.571491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.571331 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.571491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.571374 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v58dh\" (UniqueName: \"kubernetes.io/projected/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-kube-api-access-v58dh\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.571491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.571406 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.571491 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.571469 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.571734 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.571555 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.571734 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.571606 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.571734 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.571611 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.571734 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.571672 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.571734 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.571709 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.572027 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.571864 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.572027 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.571968 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.572027 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.572020 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.572469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.572450 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.573839 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.573821 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.573963 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.573937 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.628676 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.628650 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v58dh\" (UniqueName: \"kubernetes.io/projected/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-kube-api-access-v58dh\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.628828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.628683 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-dfzjt\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:52.697072 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:52.697040 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:53.123490 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:53:53.123457 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6ff2c5_461a_4dd7_8d8c_22f5f5a4264d.slice/crio-1b45bdad89f5a19ba10c4928c73654ce2eecbc2ab5408bc931f02a739c2cd64f WatchSource:0}: Error finding container 1b45bdad89f5a19ba10c4928c73654ce2eecbc2ab5408bc931f02a739c2cd64f: Status 404 returned error can't find the container with id 1b45bdad89f5a19ba10c4928c73654ce2eecbc2ab5408bc931f02a739c2cd64f Nov 03 02:53:53.127071 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:53.127027 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:53:53.127171 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:53.127124 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:53:53.127232 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:53.127166 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:53:53.127468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:53.127446 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt"] Nov 03 02:53:53.764207 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:53.764155 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" event={"ID":"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d","Type":"ContainerStarted","Data":"83432dcd187bdd3545a0d148656f90056b81f9b119ea6e62f75f0647bd862237"} Nov 03 02:53:53.764619 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:53.764231 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" event={"ID":"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d","Type":"ContainerStarted","Data":"1b45bdad89f5a19ba10c4928c73654ce2eecbc2ab5408bc931f02a739c2cd64f"} Nov 03 02:53:53.883724 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:53.883672 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" podStartSLOduration=1.8836589799999999 podStartE2EDuration="1.88365898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:53:53.880846749 +0000 UTC m=+3715.564955904" watchObservedRunningTime="2026-11-03 02:53:53.88365898 +0000 UTC m=+3715.567768134" Nov 03 02:53:53.944364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:53.942217 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:54.697402 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:54.697366 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:53:54.698899 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:54.698864 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-dfzjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 03 02:53:54.699038 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:54.698914 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" podUID="ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 03 02:53:55.698159 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:55.698121 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-dfzjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 03 02:53:55.698585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:55.698205 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" podUID="ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 03 02:53:56.697608 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:56.697570 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-dfzjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 03 02:53:56.697827 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:56.697629 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" podUID="ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 03 02:53:57.698455 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:57.698414 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-dfzjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 03 02:53:57.698976 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:57.698475 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" podUID="ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 03 02:53:58.697678 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:58.697647 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-dfzjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 03 02:53:58.697902 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:58.697702 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" podUID="ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 03 02:53:58.775115 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:58.775040 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:59.698112 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:59.698074 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-dfzjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 03 02:53:59.698347 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:53:59.698152 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" podUID="ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 03 02:54:00.698415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:00.698379 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-dfzjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 03 02:54:00.698904 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:00.698453 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" podUID="ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 03 02:54:02.080357 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:02.080321 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:54:02.082213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:02.082174 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:54:02.082900 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:02.082877 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 02:54:03.535350 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:03.535309 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:08.229333 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:08.229300 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:13.009241 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:54:13.009183 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:54:13.036816 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:13.036786 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:17.743234 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:17.743182 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:22.608588 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:22.608537 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:23.841215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:23.835767 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:24.771214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:24.770285 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:54:24.795306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:24.795258 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:54:24.830122 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:24.829777 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 02:54:25.318618 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:25.318325 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:54:25.773181 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:25.771267 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 02:54:25.773181 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:25.772054 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:54:26.028096 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:26.028018 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 02:54:27.545906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:27.545868 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:32.449443 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:32.449411 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:33.127340 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:33.127300 5463 patch_prober.go:29] interesting pod/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:54:33.127516 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:33.127367 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" podUID="bc8f16d1-4d9c-4c4b-8449-689172ef80be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:54:36.216449 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:36.216406 5463 generic.go:356] "Generic (PLEG): container finished" podID="bc8f16d1-4d9c-4c4b-8449-689172ef80be" containerID="12cf0db4a1a4f1e00c579a1aaa3e4b354df4638f8b28dd474a8915bdbf9fbff9" exitCode=0 Nov 03 02:54:36.216910 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:36.216467 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" event={"ID":"bc8f16d1-4d9c-4c4b-8449-689172ef80be","Type":"ContainerDied","Data":"12cf0db4a1a4f1e00c579a1aaa3e4b354df4638f8b28dd474a8915bdbf9fbff9"} Nov 03 02:54:36.216910 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:36.216830 5463 scope.go:122] "RemoveContainer" containerID="12cf0db4a1a4f1e00c579a1aaa3e4b354df4638f8b28dd474a8915bdbf9fbff9" Nov 03 02:54:36.218415 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:36.218381 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:54:36.218496 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:36.218462 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:54:36.218538 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:36.218495 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:54:36.859251 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:36.859220 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks"] Nov 03 02:54:36.863340 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:36.863318 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" Nov 03 02:54:36.919030 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:36.918997 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks"] Nov 03 02:54:36.919533 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:36.919506 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-dc14f6-serving-cert\"" Nov 03 02:54:37.001216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:36.994176 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2475661-edad-4611-b2bc-5930241596fa-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks\" (UID: \"c2475661-edad-4611-b2bc-5930241596fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" Nov 03 02:54:37.001216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:36.994280 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2475661-edad-4611-b2bc-5930241596fa-metrics-certs\") pod \"maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks\" (UID: \"c2475661-edad-4611-b2bc-5930241596fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" Nov 03 02:54:37.001216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:36.994371 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvp72\" (UniqueName: \"kubernetes.io/projected/c2475661-edad-4611-b2bc-5930241596fa-kube-api-access-hvp72\") pod \"maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks\" (UID: \"c2475661-edad-4611-b2bc-5930241596fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" Nov 03 02:54:37.001216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:36.994398 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2475661-edad-4611-b2bc-5930241596fa-maas-api-tls\") pod \"maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks\" (UID: \"c2475661-edad-4611-b2bc-5930241596fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" Nov 03 02:54:37.080180 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:37.080146 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:37.095825 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:37.095787 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvp72\" (UniqueName: \"kubernetes.io/projected/c2475661-edad-4611-b2bc-5930241596fa-kube-api-access-hvp72\") pod \"maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks\" (UID: \"c2475661-edad-4611-b2bc-5930241596fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" Nov 03 02:54:37.096003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:37.095838 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2475661-edad-4611-b2bc-5930241596fa-maas-api-tls\") pod \"maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks\" (UID: \"c2475661-edad-4611-b2bc-5930241596fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" Nov 03 02:54:37.096078 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:37.096061 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2475661-edad-4611-b2bc-5930241596fa-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks\" (UID: \"c2475661-edad-4611-b2bc-5930241596fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" Nov 03 02:54:37.096142 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:37.096127 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2475661-edad-4611-b2bc-5930241596fa-metrics-certs\") pod \"maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks\" (UID: \"c2475661-edad-4611-b2bc-5930241596fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" Nov 03 02:54:37.096702 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:54:37.096675 5463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-serving-cert: secret "maas-api-e2e-extmod-b-dc14f6-serving-cert" not found Nov 03 02:54:37.096910 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:54:37.096896 5463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2475661-edad-4611-b2bc-5930241596fa-maas-api-tls podName:c2475661-edad-4611-b2bc-5930241596fa nodeName:}" failed. No retries permitted until 2026-11-03 02:54:37.596870127 +0000 UTC m=+3759.280979274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2475661-edad-4611-b2bc-5930241596fa-maas-api-tls") pod "maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" (UID: "c2475661-edad-4611-b2bc-5930241596fa") : secret "maas-api-e2e-extmod-b-dc14f6-serving-cert" not found Nov 03 02:54:37.098938 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:37.098650 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2475661-edad-4611-b2bc-5930241596fa-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks\" (UID: \"c2475661-edad-4611-b2bc-5930241596fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" Nov 03 02:54:37.101153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:37.101095 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2475661-edad-4611-b2bc-5930241596fa-metrics-certs\") pod \"maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks\" (UID: \"c2475661-edad-4611-b2bc-5930241596fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" Nov 03 02:54:37.164621 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:37.164550 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvp72\" (UniqueName: \"kubernetes.io/projected/c2475661-edad-4611-b2bc-5930241596fa-kube-api-access-hvp72\") pod \"maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks\" (UID: \"c2475661-edad-4611-b2bc-5930241596fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" Nov 03 02:54:37.222106 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:37.222066 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:37.225720 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:37.225692 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" event={"ID":"bc8f16d1-4d9c-4c4b-8449-689172ef80be","Type":"ContainerStarted","Data":"20a96e641da37d8465242c1bb60de97266b7ec6ef9f0d73d2196bdaf659212e0"} Nov 03 02:54:37.382225 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:37.380938 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:37.603947 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:37.603848 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2475661-edad-4611-b2bc-5930241596fa-maas-api-tls\") pod \"maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks\" (UID: \"c2475661-edad-4611-b2bc-5930241596fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" Nov 03 02:54:37.606362 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:37.606333 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2475661-edad-4611-b2bc-5930241596fa-maas-api-tls\") pod \"maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks\" (UID: \"c2475661-edad-4611-b2bc-5930241596fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" Nov 03 02:54:37.780284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:37.780250 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" Nov 03 02:54:38.071228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:38.054534 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:54:38.071228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:38.054658 5463 patch_prober.go:29] interesting pod/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Nov 03 02:54:38.071228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:38.054700 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" podUID="bc8f16d1-4d9c-4c4b-8449-689172ef80be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Nov 03 02:54:38.185013 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:54:38.184969 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2475661_edad_4611_b2bc_5930241596fa.slice/crio-fc04d15d2fea20eafc12140b167cf87d872bbdde4ed84d2a2aa42fad5d5e7825 WatchSource:0}: Error finding container fc04d15d2fea20eafc12140b167cf87d872bbdde4ed84d2a2aa42fad5d5e7825: Status 404 returned error can't find the container with id fc04d15d2fea20eafc12140b167cf87d872bbdde4ed84d2a2aa42fad5d5e7825 Nov 03 02:54:38.185228 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:38.185202 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks"] Nov 03 02:54:38.234618 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:38.234590 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" event={"ID":"c2475661-edad-4611-b2bc-5930241596fa","Type":"ContainerStarted","Data":"fc04d15d2fea20eafc12140b167cf87d872bbdde4ed84d2a2aa42fad5d5e7825"} Nov 03 02:54:39.051540 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:39.051502 5463 patch_prober.go:29] interesting pod/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Nov 03 02:54:39.051826 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:39.051794 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" podUID="bc8f16d1-4d9c-4c4b-8449-689172ef80be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Nov 03 02:54:40.073215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:40.067536 5463 patch_prober.go:29] interesting pod/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Nov 03 02:54:40.073215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:40.067623 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" podUID="bc8f16d1-4d9c-4c4b-8449-689172ef80be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Nov 03 02:54:40.331279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:40.322612 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" event={"ID":"c2475661-edad-4611-b2bc-5930241596fa","Type":"ContainerStarted","Data":"8d189f9260a464569a2f04b0c0a5f687214c31ac84bb6c0ceb377f75feb30834"} Nov 03 02:54:40.331279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:40.323133 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" Nov 03 02:54:40.475217 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:40.474832 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:40.482157 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:40.481851 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" podStartSLOduration=3.462871019 podStartE2EDuration="4.481831851s" totalImagesPullingTime="1.018960832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:54:40.446003004 +0000 UTC m=+3762.130112160" watchObservedRunningTime="2026-11-03 02:54:40.481831851 +0000 UTC m=+3762.165941011" Nov 03 02:54:40.627268 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:40.626693 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:41.052576 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:41.052541 5463 patch_prober.go:29] interesting pod/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Nov 03 02:54:41.052862 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:41.052833 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" podUID="bc8f16d1-4d9c-4c4b-8449-689172ef80be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Nov 03 02:54:42.042210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:42.041699 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:42.056214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:42.053536 5463 patch_prober.go:29] interesting pod/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Nov 03 02:54:42.056214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:42.053591 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" podUID="bc8f16d1-4d9c-4c4b-8449-689172ef80be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Nov 03 02:54:42.629856 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:42.629811 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:43.008422 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:54:43.008387 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:54:43.054567 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:43.054539 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:54:43.054993 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:43.054973 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:54:43.055676 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:43.055659 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 02:54:43.234342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:43.234301 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:43.424690 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:43.424649 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:43.759959 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:43.759857 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:46.335363 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:46.335337 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" Nov 03 02:54:46.873829 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:46.873797 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:47.422551 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:47.422500 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:47.683547 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:47.683462 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:47.846411 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:47.846380 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:50.674592 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:50.674561 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:50.853279 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:50.853244 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:51.960722 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:51.960682 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:56.832502 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:54:56.832464 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:01.557436 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:01.557403 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:06.527547 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:06.527514 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:07.970214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:07.966851 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:08.122906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:08.122873 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:08.372270 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:08.371149 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:08.809690 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:08.809656 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:09.413321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:09.413281 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:09.605362 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:09.605268 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:11.670302 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:11.670240 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:12.513136 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.513100 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6"] Nov 03 02:55:12.627555 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.627523 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6"] Nov 03 02:55:12.627760 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.627624 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.763351 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.763264 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.763351 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.763301 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.763893 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.763404 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.763893 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.763443 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.763893 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.763461 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cls9c\" (UniqueName: \"kubernetes.io/projected/29bcbc40-df9d-489e-9a7d-50f2c49c8312-kube-api-access-cls9c\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.763893 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.763553 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.763893 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.763582 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.763893 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.763599 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.763893 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.763720 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.865203 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.865142 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.865203 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.865208 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.865457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.865278 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.865457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.865330 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.865457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.865356 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cls9c\" (UniqueName: \"kubernetes.io/projected/29bcbc40-df9d-489e-9a7d-50f2c49c8312-kube-api-access-cls9c\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.865457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.865391 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.865457 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.865430 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.865783 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.865548 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.865783 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.865597 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.865783 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.865699 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.865783 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.865712 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.865783 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.865762 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.865999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.865970 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.866480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.866457 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.867755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.867732 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.868082 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.868060 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.923164 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.923139 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.923322 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.923297 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cls9c\" (UniqueName: \"kubernetes.io/projected/29bcbc40-df9d-489e-9a7d-50f2c49c8312-kube-api-access-cls9c\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:12.968426 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:12.968401 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:13.008063 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:55:13.008033 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:55:13.509089 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:13.509052 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:13.561156 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:13.561124 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6"] Nov 03 02:55:13.591684 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:55:13.591654 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bcbc40_df9d_489e_9a7d_50f2c49c8312.slice/crio-5d87659e06b5eaf580b804b2691fbb67338ba3b6f3720bc9e26f0b083cc59f10 WatchSource:0}: Error finding container 5d87659e06b5eaf580b804b2691fbb67338ba3b6f3720bc9e26f0b083cc59f10: Status 404 returned error can't find the container with id 5d87659e06b5eaf580b804b2691fbb67338ba3b6f3720bc9e26f0b083cc59f10 Nov 03 02:55:13.594534 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:13.594497 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:13.594643 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:13.594573 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:13.594643 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:13.594602 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:13.660549 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:13.660517 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" event={"ID":"29bcbc40-df9d-489e-9a7d-50f2c49c8312","Type":"ContainerStarted","Data":"5d87659e06b5eaf580b804b2691fbb67338ba3b6f3720bc9e26f0b083cc59f10"} Nov 03 02:55:13.733124 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:13.733089 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:13.944654 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:13.944614 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:14.688946 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:14.688883 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" event={"ID":"29bcbc40-df9d-489e-9a7d-50f2c49c8312","Type":"ContainerStarted","Data":"a0f81cfa8976b39a2e878a2be98d7af948c67b730420ad1acffcbca92c0e2faa"} Nov 03 02:55:14.867627 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:14.867566 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" podStartSLOduration=2.867546999 podStartE2EDuration="2.867546999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:55:14.857847005 +0000 UTC m=+3796.541956161" watchObservedRunningTime="2026-11-03 02:55:14.867546999 +0000 UTC m=+3796.551656156" Nov 03 02:55:14.971220 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:14.970022 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:14.980685 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:14.980488 5463 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Nov 03 02:55:14.980685 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:14.980544 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" podUID="29bcbc40-df9d-489e-9a7d-50f2c49c8312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Nov 03 02:55:15.474213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:15.473312 5463 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:55:15.474213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:15.473393 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" podUID="3b38f888-2173-4d9b-a4cb-047d5086e894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:55:15.973586 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:15.973551 5463 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Nov 03 02:55:15.974170 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:15.974140 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" podUID="29bcbc40-df9d-489e-9a7d-50f2c49c8312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Nov 03 02:55:16.807130 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:16.807082 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:16.997203 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:16.992173 5463 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Nov 03 02:55:17.013083 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.012589 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" podUID="29bcbc40-df9d-489e-9a7d-50f2c49c8312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Nov 03 02:55:17.013083 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.010338 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:17.184793 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.183995 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:17.344207 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.344164 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6"] Nov 03 02:55:17.380598 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.378634 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6" Nov 03 02:55:17.402482 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.402444 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6"] Nov 03 02:55:17.449473 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.449394 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ddcbd0\"" Nov 03 02:55:17.450153 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.450129 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ddcbd0-dockercfg-s45jf\"" Nov 03 02:55:17.608046 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.604395 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc52833c-b2ac-4cde-96e7-6b52dd86a747-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6\" (UID: \"dc52833c-b2ac-4cde-96e7-6b52dd86a747\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6" Nov 03 02:55:17.608046 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.604444 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcx6d\" (UniqueName: \"kubernetes.io/projected/dc52833c-b2ac-4cde-96e7-6b52dd86a747-kube-api-access-jcx6d\") pod \"payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6\" (UID: \"dc52833c-b2ac-4cde-96e7-6b52dd86a747\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6" Nov 03 02:55:17.615029 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.614061 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v"] Nov 03 02:55:17.690224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.689379 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v"] Nov 03 02:55:17.690224 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.689575 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v" Nov 03 02:55:17.705674 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.705601 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc52833c-b2ac-4cde-96e7-6b52dd86a747-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6\" (UID: \"dc52833c-b2ac-4cde-96e7-6b52dd86a747\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6" Nov 03 02:55:17.705797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.705712 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcx6d\" (UniqueName: \"kubernetes.io/projected/dc52833c-b2ac-4cde-96e7-6b52dd86a747-kube-api-access-jcx6d\") pod \"payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6\" (UID: \"dc52833c-b2ac-4cde-96e7-6b52dd86a747\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6" Nov 03 02:55:17.706489 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.706467 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc52833c-b2ac-4cde-96e7-6b52dd86a747-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6\" (UID: \"dc52833c-b2ac-4cde-96e7-6b52dd86a747\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6" Nov 03 02:55:17.771214 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.771149 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcx6d\" (UniqueName: \"kubernetes.io/projected/dc52833c-b2ac-4cde-96e7-6b52dd86a747-kube-api-access-jcx6d\") pod \"payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6\" (UID: \"dc52833c-b2ac-4cde-96e7-6b52dd86a747\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6" Nov 03 02:55:17.807990 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.807909 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf418c5-21e9-4424-81a9-e928cb205a4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v\" (UID: \"1bf418c5-21e9-4424-81a9-e928cb205a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v" Nov 03 02:55:17.808184 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.808083 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pcnt\" (UniqueName: \"kubernetes.io/projected/1bf418c5-21e9-4424-81a9-e928cb205a4b-kube-api-access-5pcnt\") pod \"payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v\" (UID: \"1bf418c5-21e9-4424-81a9-e928cb205a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v" Nov 03 02:55:17.909420 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.909376 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf418c5-21e9-4424-81a9-e928cb205a4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v\" (UID: \"1bf418c5-21e9-4424-81a9-e928cb205a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v" Nov 03 02:55:17.909617 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.909469 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pcnt\" (UniqueName: \"kubernetes.io/projected/1bf418c5-21e9-4424-81a9-e928cb205a4b-kube-api-access-5pcnt\") pod \"payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v\" (UID: \"1bf418c5-21e9-4424-81a9-e928cb205a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v" Nov 03 02:55:17.910142 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.910119 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf418c5-21e9-4424-81a9-e928cb205a4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v\" (UID: \"1bf418c5-21e9-4424-81a9-e928cb205a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v" Nov 03 02:55:17.971288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.970521 5463 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Nov 03 02:55:17.971288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.970581 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" podUID="29bcbc40-df9d-489e-9a7d-50f2c49c8312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Nov 03 02:55:17.994246 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:17.981121 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pcnt\" (UniqueName: \"kubernetes.io/projected/1bf418c5-21e9-4424-81a9-e928cb205a4b-kube-api-access-5pcnt\") pod \"payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v\" (UID: \"1bf418c5-21e9-4424-81a9-e928cb205a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v" Nov 03 02:55:18.043100 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:18.043065 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v" Nov 03 02:55:18.045204 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:18.045172 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6" Nov 03 02:55:18.535002 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:18.533327 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v"] Nov 03 02:55:18.590658 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:55:18.590624 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc52833c_b2ac_4cde_96e7_6b52dd86a747.slice/crio-e2a290133e7a6f436b85c601b8f6cabe23c92436aadafe1955a9787bea303854 WatchSource:0}: Error finding container e2a290133e7a6f436b85c601b8f6cabe23c92436aadafe1955a9787bea303854: Status 404 returned error can't find the container with id e2a290133e7a6f436b85c601b8f6cabe23c92436aadafe1955a9787bea303854 Nov 03 02:55:18.592213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:18.592059 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6"] Nov 03 02:55:18.731321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:18.731283 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6" event={"ID":"dc52833c-b2ac-4cde-96e7-6b52dd86a747","Type":"ContainerStarted","Data":"e2a290133e7a6f436b85c601b8f6cabe23c92436aadafe1955a9787bea303854"} Nov 03 02:55:18.732781 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:18.732752 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v" event={"ID":"1bf418c5-21e9-4424-81a9-e928cb205a4b","Type":"ContainerStarted","Data":"3d04334efbb8ff5d9a3e39bf9808dd9f4c70c2c23b704da84997e5f7a089a226"} Nov 03 02:55:18.984215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:18.983770 5463 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Nov 03 02:55:18.984215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:18.983846 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" podUID="29bcbc40-df9d-489e-9a7d-50f2c49c8312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Nov 03 02:55:19.742381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:19.742343 5463 generic.go:356] "Generic (PLEG): container finished" podID="3b38f888-2173-4d9b-a4cb-047d5086e894" containerID="66e900090ec0db270731d1f9c0a91e33778a6d64c6f796dbb40a6cdb762dce19" exitCode=0 Nov 03 02:55:19.742381 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:19.742422 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" event={"ID":"3b38f888-2173-4d9b-a4cb-047d5086e894","Type":"ContainerDied","Data":"66e900090ec0db270731d1f9c0a91e33778a6d64c6f796dbb40a6cdb762dce19"} Nov 03 02:55:19.743211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:19.742920 5463 scope.go:122] "RemoveContainer" containerID="66e900090ec0db270731d1f9c0a91e33778a6d64c6f796dbb40a6cdb762dce19" Nov 03 02:55:19.744671 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:19.744641 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:19.744749 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:19.744713 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:19.744787 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:19.744746 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:19.745364 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:19.745338 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6" event={"ID":"dc52833c-b2ac-4cde-96e7-6b52dd86a747","Type":"ContainerStarted","Data":"b143abf65a0fcb98e6bbd2703f1d26e0617e22f2b0537a5a87848196db8537e2"} Nov 03 02:55:19.745460 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:19.745432 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6" Nov 03 02:55:19.761212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:19.758316 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v" event={"ID":"1bf418c5-21e9-4424-81a9-e928cb205a4b","Type":"ContainerStarted","Data":"b9077b5a7f740f7abfda16b1f1246a432bd64ab4010f19cf1fb0f0689b73e487"} Nov 03 02:55:19.761212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:19.758478 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v" Nov 03 02:55:19.987221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:19.985159 5463 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Nov 03 02:55:19.987221 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:19.985233 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" podUID="29bcbc40-df9d-489e-9a7d-50f2c49c8312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Nov 03 02:55:20.029870 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:20.029815 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v" podStartSLOduration=3.029800677 podStartE2EDuration="3.029800677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:55:20.028646713 +0000 UTC m=+3801.712755865" watchObservedRunningTime="2026-11-03 02:55:20.029800677 +0000 UTC m=+3801.713909832" Nov 03 02:55:20.156791 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:20.156712 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6" podStartSLOduration=3.156694102 podStartE2EDuration="3.156694102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:55:20.143521693 +0000 UTC m=+3801.827630860" watchObservedRunningTime="2026-11-03 02:55:20.156694102 +0000 UTC m=+3801.840803256" Nov 03 02:55:20.787120 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:20.787078 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" event={"ID":"3b38f888-2173-4d9b-a4cb-047d5086e894","Type":"ContainerStarted","Data":"063f83d315206498c945de2c3c9d040232bf2c41e159e00558294db6967123d4"} Nov 03 02:55:20.883218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:20.879561 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:55:20.883218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:20.879681 5463 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body= Nov 03 02:55:20.883218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:20.879727 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" podUID="3b38f888-2173-4d9b-a4cb-047d5086e894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" Nov 03 02:55:21.143519 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:21.143489 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:21.144004 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:21.143971 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:21.146540 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:21.146514 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 02:55:21.736253 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:21.736219 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:21.879373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:21.878560 5463 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body= Nov 03 02:55:21.879373 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:21.878628 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" podUID="3b38f888-2173-4d9b-a4cb-047d5086e894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" Nov 03 02:55:22.884289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:22.877499 5463 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body= Nov 03 02:55:22.884289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:22.877560 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" podUID="3b38f888-2173-4d9b-a4cb-047d5086e894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" Nov 03 02:55:23.878216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:23.877572 5463 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body= Nov 03 02:55:23.878216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:23.877636 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" podUID="3b38f888-2173-4d9b-a4cb-047d5086e894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" Nov 03 02:55:24.036585 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:24.036534 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:24.875950 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:24.875513 5463 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body= Nov 03 02:55:24.875950 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:24.875571 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" podUID="3b38f888-2173-4d9b-a4cb-047d5086e894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" Nov 03 02:55:25.875722 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:25.875678 5463 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body= Nov 03 02:55:25.876162 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:25.875752 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" podUID="3b38f888-2173-4d9b-a4cb-047d5086e894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" Nov 03 02:55:26.879869 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:26.879839 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:55:26.880335 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:26.880316 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:55:26.881125 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:26.881096 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 02:55:26.881289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:26.881256 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:27.385238 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:27.385181 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:27.864006 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:27.863968 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:28.496271 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.496227 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd"] Nov 03 02:55:28.500554 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.500534 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.575211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.565106 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9a5e31-openshift-default-dockercfg-xw6p6\"" Nov 03 02:55:28.673052 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.672953 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:28.674236 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.674107 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-podinfo\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.674614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.674590 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-workload-socket\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.674824 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.674805 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istiod-ca-cert\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.674987 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.674972 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-workload-certs\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.675179 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.675162 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-data\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.675388 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.675372 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-credential-socket\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.675566 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.675551 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-envoy\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.675736 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.675721 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-token\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.675947 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.675931 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx2mp\" (UniqueName: \"kubernetes.io/projected/0d32bca4-00c3-4968-92b0-86c8ed0738c9-kube-api-access-zx2mp\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.685631 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.684264 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd"] Nov 03 02:55:28.777485 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.777398 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-token\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.777681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.777490 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx2mp\" (UniqueName: \"kubernetes.io/projected/0d32bca4-00c3-4968-92b0-86c8ed0738c9-kube-api-access-zx2mp\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.777681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.777583 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-podinfo\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.777817 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.777700 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-workload-socket\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.777817 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.777736 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istiod-ca-cert\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.777817 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.777774 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-workload-certs\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.778016 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.777838 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-data\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.778016 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.777877 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-credential-socket\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.778016 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.777934 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-envoy\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.778341 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.778313 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-workload-certs\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.778469 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.778377 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-credential-socket\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.778746 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.778719 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istiod-ca-cert\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.778882 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.778864 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-workload-socket\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.778998 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.778978 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-data\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.780707 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.780682 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-envoy\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.781037 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.781019 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-podinfo\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.834574 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.834553 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-token\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.834771 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.834755 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx2mp\" (UniqueName: \"kubernetes.io/projected/0d32bca4-00c3-4968-92b0-86c8ed0738c9-kube-api-access-zx2mp\") pod \"e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:28.909964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:28.909921 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:29.113873 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:29.113784 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:29.545714 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:29.545684 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd"] Nov 03 02:55:29.583597 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:55:29.583557 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d32bca4_00c3_4968_92b0_86c8ed0738c9.slice/crio-89e1f9caa9a9698451add4221564f371de56bd042108a8025b91ccb74425fca0 WatchSource:0}: Error finding container 89e1f9caa9a9698451add4221564f371de56bd042108a8025b91ccb74425fca0: Status 404 returned error can't find the container with id 89e1f9caa9a9698451add4221564f371de56bd042108a8025b91ccb74425fca0 Nov 03 02:55:29.586416 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:29.586373 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:29.586524 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:29.586463 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:29.586524 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:29.586504 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:29.875009 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:29.874924 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" event={"ID":"0d32bca4-00c3-4968-92b0-86c8ed0738c9","Type":"ContainerStarted","Data":"6519998f7c7b3968fb6b5eba6273f26ae8d70dc57813b4635673e6e1a271e40d"} Nov 03 02:55:29.875009 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:29.874992 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" event={"ID":"0d32bca4-00c3-4968-92b0-86c8ed0738c9","Type":"ContainerStarted","Data":"89e1f9caa9a9698451add4221564f371de56bd042108a8025b91ccb74425fca0"} Nov 03 02:55:29.994260 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:29.994209 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" podStartSLOduration=1.994175359 podStartE2EDuration="1.994175359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:55:29.991798331 +0000 UTC m=+3811.675907489" watchObservedRunningTime="2026-11-03 02:55:29.994175359 +0000 UTC m=+3811.678284514" Nov 03 02:55:30.114807 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:30.114776 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:30.116657 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:30.116626 5463 patch_prober.go:29] interesting pod/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body= Nov 03 02:55:30.116793 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:30.116688 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" podUID="0d32bca4-00c3-4968-92b0-86c8ed0738c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" Nov 03 02:55:30.791269 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:30.791129 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v" Nov 03 02:55:30.792468 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:30.792444 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6" Nov 03 02:55:31.118884 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:31.118856 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:31.872008 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:31.871975 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:31.904596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:31.904567 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:31.906601 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:31.906567 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:55:32.872284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:32.872248 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:33.042410 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:33.042377 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:33.303964 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:33.303932 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:36.082049 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:36.082013 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:36.255641 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:36.255590 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:37.198072 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:37.198035 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:42.041129 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.041059 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:42.755298 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.755266 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2"] Nov 03 02:55:42.759213 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.759180 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.830222 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.830176 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.830360 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.830310 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.830360 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.830341 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.830443 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.830430 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.830480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.830458 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.830529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.830491 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fchl4\" (UniqueName: \"kubernetes.io/projected/7aa45c20-0b65-4808-80e2-ba8c767966f9-kube-api-access-fchl4\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.830600 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.830587 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.830647 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.830607 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa45c20-0b65-4808-80e2-ba8c767966f9-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.830706 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.830659 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.931492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.931458 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.931492 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.931491 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa45c20-0b65-4808-80e2-ba8c767966f9-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.931711 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.931532 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.931711 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.931605 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.931711 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.931677 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.931711 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.931696 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.931950 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.931769 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.931950 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.931814 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.931950 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.931862 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fchl4\" (UniqueName: \"kubernetes.io/projected/7aa45c20-0b65-4808-80e2-ba8c767966f9-kube-api-access-fchl4\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.932166 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.932121 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.932284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.932181 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.932284 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.932251 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.932412 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.932340 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa45c20-0b65-4808-80e2-ba8c767966f9-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.932539 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.932518 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.934303 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.934280 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.934406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.934306 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.954705 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.954680 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2"] Nov 03 02:55:42.989701 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.989638 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fchl4\" (UniqueName: \"kubernetes.io/projected/7aa45c20-0b65-4808-80e2-ba8c767966f9-kube-api-access-fchl4\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:42.993032 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:42.992977 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:43.014849 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:55:43.014783 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:55:43.074034 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:43.074008 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:43.495722 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:55:43.495694 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa45c20_0b65_4808_80e2_ba8c767966f9.slice/crio-d89beba393621fcaa6c3125e48ed8b6a8397ab332eb745de134325169fd27472 WatchSource:0}: Error finding container d89beba393621fcaa6c3125e48ed8b6a8397ab332eb745de134325169fd27472: Status 404 returned error can't find the container with id d89beba393621fcaa6c3125e48ed8b6a8397ab332eb745de134325169fd27472 Nov 03 02:55:43.498731 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:43.498697 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:43.498824 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:43.498786 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:43.498886 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:43.498823 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:55:43.498886 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:43.498848 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2"] Nov 03 02:55:44.003839 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:44.003799 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" event={"ID":"7aa45c20-0b65-4808-80e2-ba8c767966f9","Type":"ContainerStarted","Data":"d07472982e7835970e7ba007cc30c54df7bf689f5def87d47179f62bd6dbb0d9"} Nov 03 02:55:44.004047 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:44.003871 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" event={"ID":"7aa45c20-0b65-4808-80e2-ba8c767966f9","Type":"ContainerStarted","Data":"d89beba393621fcaa6c3125e48ed8b6a8397ab332eb745de134325169fd27472"} Nov 03 02:55:44.074634 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:44.074594 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:44.076814 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:44.076779 5463 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Nov 03 02:55:44.076963 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:44.076828 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" podUID="7aa45c20-0b65-4808-80e2-ba8c767966f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Nov 03 02:55:44.231390 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:44.231332 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" podStartSLOduration=2.23131951 podStartE2EDuration="2.23131951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:55:44.183464948 +0000 UTC m=+3825.867574105" watchObservedRunningTime="2026-11-03 02:55:44.23131951 +0000 UTC m=+3825.915428665" Nov 03 02:55:45.075597 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:45.075556 5463 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Nov 03 02:55:45.076259 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:45.076224 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" podUID="7aa45c20-0b65-4808-80e2-ba8c767966f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Nov 03 02:55:46.079609 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:46.079572 5463 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Nov 03 02:55:46.080016 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:46.079638 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" podUID="7aa45c20-0b65-4808-80e2-ba8c767966f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Nov 03 02:55:46.881241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:46.881185 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:47.077529 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:47.077494 5463 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Nov 03 02:55:47.077797 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:47.077772 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" podUID="7aa45c20-0b65-4808-80e2-ba8c767966f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Nov 03 02:55:48.079521 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:48.079484 5463 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Nov 03 02:55:48.080596 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:48.080550 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" podUID="7aa45c20-0b65-4808-80e2-ba8c767966f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Nov 03 02:55:49.074593 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:49.074555 5463 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Nov 03 02:55:49.074901 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:49.074872 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" podUID="7aa45c20-0b65-4808-80e2-ba8c767966f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Nov 03 02:55:50.081321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:50.081291 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:50.081774 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:50.081752 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:50.082653 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:50.082627 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 02:55:51.603893 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:51.603856 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:51.806175 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:51.806110 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:56.303850 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:55:56.303813 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:01.025709 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:01.025653 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:05.804933 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:05.804892 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:10.632549 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:10.632514 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:13.009299 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:56:13.009265 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:56:14.772216 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:14.772156 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:14.918294 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:14.918253 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:15.558690 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:15.558652 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:15.735700 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:15.735665 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:15.884342 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:15.884249 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:20.536608 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:20.536573 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:20.752295 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:20.752254 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:22.283590 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:22.283486 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:22.437329 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:22.437289 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:24.349140 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:24.349106 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:25.763430 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:25.763393 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:26.347777 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:26.347746 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:30.528440 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:30.528403 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:35.382235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:35.382183 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:40.148116 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:40.148016 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:43.008003 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:56:43.007968 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:56:44.823285 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:44.823250 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:49.461525 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:49.461487 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:49.621635 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:49.621602 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:49.773614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:49.773517 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:54.413517 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:54.413483 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:58.804872 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:58.804832 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:58.948249 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:58.948186 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:59.173595 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:56:59.173565 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:03.139887 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:03.139857 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:57:03.142352 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:03.142327 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 02:57:03.956895 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:03.956862 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:08.359586 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:08.359548 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd"] Nov 03 02:57:08.364045 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:08.364026 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" Nov 03 02:57:08.417841 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:08.417811 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd"] Nov 03 02:57:08.462618 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:08.462584 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9a5e31-serving-cert\"" Nov 03 02:57:08.472461 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:08.472422 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-maas-api-tls\") pod \"maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd\" (UID: \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" Nov 03 02:57:08.472661 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:08.472642 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2c6l\" (UniqueName: \"kubernetes.io/projected/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-kube-api-access-k2c6l\") pod \"maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd\" (UID: \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" Nov 03 02:57:08.472795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:08.472776 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-metrics-certs\") pod \"maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd\" (UID: \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" Nov 03 02:57:08.472842 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:08.472832 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd\" (UID: \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" Nov 03 02:57:08.574052 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:08.574011 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-metrics-certs\") pod \"maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd\" (UID: \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" Nov 03 02:57:08.574301 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:08.574059 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd\" (UID: \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" Nov 03 02:57:08.574301 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:08.574110 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-maas-api-tls\") pod \"maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd\" (UID: \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" Nov 03 02:57:08.574439 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:57:08.574290 5463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-serving-cert: secret "maas-api-e2e-inf-b-9a5e31-serving-cert" not found Nov 03 02:57:08.574439 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:08.574351 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2c6l\" (UniqueName: \"kubernetes.io/projected/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-kube-api-access-k2c6l\") pod \"maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd\" (UID: \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" Nov 03 02:57:08.574439 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:57:08.574382 5463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-maas-api-tls podName:bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e nodeName:}" failed. No retries permitted until 2026-11-03 02:57:09.074360444 +0000 UTC m=+3910.758469578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-maas-api-tls") pod "maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" (UID: "bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e") : secret "maas-api-e2e-inf-b-9a5e31-serving-cert" not found Nov 03 02:57:08.575209 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:08.575165 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd\" (UID: \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" Nov 03 02:57:08.577073 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:08.577050 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-metrics-certs\") pod \"maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd\" (UID: \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" Nov 03 02:57:08.578575 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:08.578495 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:08.642263 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:08.641888 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2c6l\" (UniqueName: \"kubernetes.io/projected/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-kube-api-access-k2c6l\") pod \"maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd\" (UID: \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" Nov 03 02:57:08.744321 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:08.744117 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:08.965005 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:08.964970 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:09.092733 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:09.092698 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-maas-api-tls\") pod \"maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd\" (UID: \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" Nov 03 02:57:09.095247 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:09.095219 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-maas-api-tls\") pod \"maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd\" (UID: \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" Nov 03 02:57:09.279003 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:09.278926 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" Nov 03 02:57:09.659950 ip-10-0-137-69 kubenswrapper[5463]: W1103 02:57:09.659923 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8ec8c4_a60c_48e3_8e8c_9ff53af84a5e.slice/crio-e9b2ca2a79b3f4f59239201204ef08c32e81151c890ea1aed1195fd69732dd7c WatchSource:0}: Error finding container e9b2ca2a79b3f4f59239201204ef08c32e81151c890ea1aed1195fd69732dd7c: Status 404 returned error can't find the container with id e9b2ca2a79b3f4f59239201204ef08c32e81151c890ea1aed1195fd69732dd7c Nov 03 02:57:09.664867 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:09.664836 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd"] Nov 03 02:57:09.804447 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:09.804414 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" event={"ID":"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e","Type":"ContainerStarted","Data":"e9b2ca2a79b3f4f59239201204ef08c32e81151c890ea1aed1195fd69732dd7c"} Nov 03 02:57:10.812861 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:10.812771 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" event={"ID":"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e","Type":"ContainerStarted","Data":"48219a1e4243f24aca80055c384540cf2cb01a102e390bed002804b0c6ba642b"} Nov 03 02:57:10.813261 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:10.812998 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" Nov 03 02:57:10.928753 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:10.928706 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" podStartSLOduration=2.116865535 podStartE2EDuration="2.928693082s" totalImagesPullingTime="811.827547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:57:10.927256303 +0000 UTC m=+3912.611365458" watchObservedRunningTime="2026-11-03 02:57:10.928693082 +0000 UTC m=+3912.612802242" Nov 03 02:57:13.008302 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:57:13.008262 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:57:13.521575 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:13.521541 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:16.826575 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:16.826547 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" Nov 03 02:57:18.231839 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:18.231801 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:23.049092 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:23.049051 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:24.671681 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:24.671648 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:27.715442 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:27.715397 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:31.897715 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:31.897680 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:33.611999 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:33.611964 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:38.334429 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:38.334397 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:40.644646 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:40.644614 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:40.804767 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:40.804734 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:42.234064 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:42.234020 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:42.860414 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:42.860377 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:43.094456 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:57:43.094417 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:57:43.180609 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:43.180566 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:43.697553 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:43.697518 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:46.631948 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:46.631911 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:48.478438 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:48.477484 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:49.694229 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:49.694171 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:50.982484 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:50.982445 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:51.135138 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:51.135083 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:52.537162 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:52.537111 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:52.894691 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:52.894653 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:53.391869 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:53.391780 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:53.544494 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:53.544451 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:54.831023 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:54.830984 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:55.294786 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:55.294742 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:57.336300 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:57.336271 5463 generic.go:356] "Generic (PLEG): container finished" podID="0d32bca4-00c3-4968-92b0-86c8ed0738c9" containerID="6519998f7c7b3968fb6b5eba6273f26ae8d70dc57813b4635673e6e1a271e40d" exitCode=0 Nov 03 02:57:57.336722 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:57.336324 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" event={"ID":"0d32bca4-00c3-4968-92b0-86c8ed0738c9","Type":"ContainerDied","Data":"6519998f7c7b3968fb6b5eba6273f26ae8d70dc57813b4635673e6e1a271e40d"} Nov 03 02:57:57.336722 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:57.336684 5463 scope.go:122] "RemoveContainer" containerID="6519998f7c7b3968fb6b5eba6273f26ae8d70dc57813b4635673e6e1a271e40d" Nov 03 02:57:57.338384 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:57.338350 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:57:57.338497 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:57.338424 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:57:57.338497 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:57.338456 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:57:58.244990 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:58.244956 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:58.355541 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:58.355501 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" event={"ID":"0d32bca4-00c3-4968-92b0-86c8ed0738c9","Type":"ContainerStarted","Data":"75019cdcba71b15789bb57ac75182e113a8368fa207e0db74d381a921bf226c4"} Nov 03 02:57:58.397719 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:58.397670 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:59.113906 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:59.113875 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:57:59.115061 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:59.115029 5463 patch_prober.go:29] interesting pod/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body= Nov 03 02:57:59.115177 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:59.115089 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" podUID="0d32bca4-00c3-4968-92b0-86c8ed0738c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" Nov 03 02:57:59.769215 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:59.768025 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:59.921828 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:57:59.921789 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:00.115223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:00.115113 5463 patch_prober.go:29] interesting pod/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body= Nov 03 02:58:00.115223 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:00.115168 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" podUID="0d32bca4-00c3-4968-92b0-86c8ed0738c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" Nov 03 02:58:00.244458 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:00.244426 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:01.114145 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:01.114113 5463 patch_prober.go:29] interesting pod/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body= Nov 03 02:58:01.114595 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:01.114170 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" podUID="0d32bca4-00c3-4968-92b0-86c8ed0738c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" Nov 03 02:58:01.904617 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:01.904588 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:58:02.114861 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:02.114828 5463 patch_prober.go:29] interesting pod/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body= Nov 03 02:58:02.115339 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:02.114894 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" podUID="0d32bca4-00c3-4968-92b0-86c8ed0738c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" Nov 03 02:58:03.015113 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:03.015065 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:03.115014 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:03.114983 5463 patch_prober.go:29] interesting pod/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body= Nov 03 02:58:03.115483 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:03.115047 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" podUID="0d32bca4-00c3-4968-92b0-86c8ed0738c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" Nov 03 02:58:03.173682 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:03.173644 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:03.419998 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:03.419965 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:04.119146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:04.119117 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:58:04.120283 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:04.120260 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 02:58:04.983340 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:04.983302 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:09.821687 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:09.821595 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:12.872940 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:12.872884 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:13.008803 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:58:13.008765 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:58:14.527308 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:14.527272 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:18.584581 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:18.584537 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:19.286210 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:19.286165 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:19.551664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:19.551583 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc"] Nov 03 02:58:19.822418 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:19.822331 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:20.014422 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:20.014375 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:24.146633 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:24.146583 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:24.552388 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:24.552306 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc" podUID="859202a1-dd87-427e-8e59-acfe73d4b6d8" containerName="payload-pre-processing" containerID="cri-o://8d52c31e4d9d3bdbc4a134742d43935b77446382e1e3f2036c3d07972032ad77" gracePeriod=25 Nov 03 02:58:24.660643 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:24.660576 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:24.877961 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:24.877931 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc" Nov 03 02:58:24.946561 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:24.946529 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/859202a1-dd87-427e-8e59-acfe73d4b6d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859202a1-dd87-427e-8e59-acfe73d4b6d8-plugins-config-volume\") pod \"859202a1-dd87-427e-8e59-acfe73d4b6d8\" (UID: \"859202a1-dd87-427e-8e59-acfe73d4b6d8\") " Nov 03 02:58:24.946763 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:24.946624 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/859202a1-dd87-427e-8e59-acfe73d4b6d8-kube-api-access-vhjq9\" (UniqueName: \"kubernetes.io/projected/859202a1-dd87-427e-8e59-acfe73d4b6d8-kube-api-access-vhjq9\") pod \"859202a1-dd87-427e-8e59-acfe73d4b6d8\" (UID: \"859202a1-dd87-427e-8e59-acfe73d4b6d8\") " Nov 03 02:58:24.947058 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:24.947026 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/859202a1-dd87-427e-8e59-acfe73d4b6d8-plugins-config-volume" pod "859202a1-dd87-427e-8e59-acfe73d4b6d8" (UID: "859202a1-dd87-427e-8e59-acfe73d4b6d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:58:24.947407 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:24.947379 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859202a1-dd87-427e-8e59-acfe73d4b6d8-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:24.949053 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:24.949022 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859202a1-dd87-427e-8e59-acfe73d4b6d8-kube-api-access-vhjq9" pod "859202a1-dd87-427e-8e59-acfe73d4b6d8" (UID: "859202a1-dd87-427e-8e59-acfe73d4b6d8"). InnerVolumeSpecName "kube-api-access-vhjq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:58:25.006672 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:25.006633 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:25.048724 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:25.048690 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhjq9\" (UniqueName: \"kubernetes.io/projected/859202a1-dd87-427e-8e59-acfe73d4b6d8-kube-api-access-vhjq9\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:25.662146 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:25.662113 5463 generic.go:356] "Generic (PLEG): container finished" podID="859202a1-dd87-427e-8e59-acfe73d4b6d8" containerID="8d52c31e4d9d3bdbc4a134742d43935b77446382e1e3f2036c3d07972032ad77" exitCode=0 Nov 03 02:58:25.662708 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:25.662156 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc" Nov 03 02:58:25.662708 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:25.662208 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc" event={"ID":"859202a1-dd87-427e-8e59-acfe73d4b6d8","Type":"ContainerDied","Data":"8d52c31e4d9d3bdbc4a134742d43935b77446382e1e3f2036c3d07972032ad77"} Nov 03 02:58:25.662708 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:25.662250 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc" event={"ID":"859202a1-dd87-427e-8e59-acfe73d4b6d8","Type":"ContainerDied","Data":"fc9cee867ea41130db12f7aa6cb78c651645da890375e5b3fb0f3f5aa1246f7f"} Nov 03 02:58:25.662708 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:25.662271 5463 scope.go:122] "RemoveContainer" containerID="8d52c31e4d9d3bdbc4a134742d43935b77446382e1e3f2036c3d07972032ad77" Nov 03 02:58:25.674472 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:25.674421 5463 scope.go:122] "RemoveContainer" containerID="8d52c31e4d9d3bdbc4a134742d43935b77446382e1e3f2036c3d07972032ad77" Nov 03 02:58:25.674741 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:58:25.674723 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d52c31e4d9d3bdbc4a134742d43935b77446382e1e3f2036c3d07972032ad77\": container with ID starting with 8d52c31e4d9d3bdbc4a134742d43935b77446382e1e3f2036c3d07972032ad77 not found: ID does not exist" containerID="8d52c31e4d9d3bdbc4a134742d43935b77446382e1e3f2036c3d07972032ad77" Nov 03 02:58:25.674795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:25.674752 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d52c31e4d9d3bdbc4a134742d43935b77446382e1e3f2036c3d07972032ad77"} err="failed to get container status \"8d52c31e4d9d3bdbc4a134742d43935b77446382e1e3f2036c3d07972032ad77\": rpc error: code = NotFound desc = could not find container \"8d52c31e4d9d3bdbc4a134742d43935b77446382e1e3f2036c3d07972032ad77\": container with ID starting with 8d52c31e4d9d3bdbc4a134742d43935b77446382e1e3f2036c3d07972032ad77 not found: ID does not exist" Nov 03 02:58:25.830715 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:25.830685 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc"] Nov 03 02:58:25.880664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:25.880635 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-95152a-68c975dc79-4hwnc"] Nov 03 02:58:26.446084 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:26.446049 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="859202a1-dd87-427e-8e59-acfe73d4b6d8" path="/var/lib/kubelet/pods/859202a1-dd87-427e-8e59-acfe73d4b6d8/volumes" Nov 03 02:58:29.621889 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:29.621857 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:32.022802 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:32.022768 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:34.695572 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:34.695523 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:38.032686 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:38.032652 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:38.623855 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:38.623820 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9"] Nov 03 02:58:38.882298 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:38.882221 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:39.032006 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:39.031972 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:39.757296 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:39.757259 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:43.008578 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:58:43.008533 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:58:43.558483 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:43.558434 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:43.624316 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:43.624273 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9" podUID="a032506e-d4ff-49ea-a5e2-a1fa2137b3d4" containerName="payload-pre-processing" containerID="cri-o://0792f9df29ecc4a52e4e5ef1d9e5b8b6695647a8ae2b4a2afa61791ac3f3ae7a" gracePeriod=25 Nov 03 02:58:43.814232 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:43.814020 5463 generic.go:356] "Generic (PLEG): container finished" podID="a032506e-d4ff-49ea-a5e2-a1fa2137b3d4" containerID="0792f9df29ecc4a52e4e5ef1d9e5b8b6695647a8ae2b4a2afa61791ac3f3ae7a" exitCode=0 Nov 03 02:58:43.814232 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:43.814106 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9" event={"ID":"a032506e-d4ff-49ea-a5e2-a1fa2137b3d4","Type":"ContainerDied","Data":"0792f9df29ecc4a52e4e5ef1d9e5b8b6695647a8ae2b4a2afa61791ac3f3ae7a"} Nov 03 02:58:43.922055 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:43.922031 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9" Nov 03 02:58:44.030689 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:44.030650 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a032506e-d4ff-49ea-a5e2-a1fa2137b3d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a032506e-d4ff-49ea-a5e2-a1fa2137b3d4-plugins-config-volume\") pod \"a032506e-d4ff-49ea-a5e2-a1fa2137b3d4\" (UID: \"a032506e-d4ff-49ea-a5e2-a1fa2137b3d4\") " Nov 03 02:58:44.031230 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:44.030919 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a032506e-d4ff-49ea-a5e2-a1fa2137b3d4-kube-api-access-gn7vd\" (UniqueName: \"kubernetes.io/projected/a032506e-d4ff-49ea-a5e2-a1fa2137b3d4-kube-api-access-gn7vd\") pod \"a032506e-d4ff-49ea-a5e2-a1fa2137b3d4\" (UID: \"a032506e-d4ff-49ea-a5e2-a1fa2137b3d4\") " Nov 03 02:58:44.031513 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:44.031484 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a032506e-d4ff-49ea-a5e2-a1fa2137b3d4-plugins-config-volume" pod "a032506e-d4ff-49ea-a5e2-a1fa2137b3d4" (UID: "a032506e-d4ff-49ea-a5e2-a1fa2137b3d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:58:44.031903 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:44.031877 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a032506e-d4ff-49ea-a5e2-a1fa2137b3d4-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:44.033926 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:44.033890 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a032506e-d4ff-49ea-a5e2-a1fa2137b3d4-kube-api-access-gn7vd" pod "a032506e-d4ff-49ea-a5e2-a1fa2137b3d4" (UID: "a032506e-d4ff-49ea-a5e2-a1fa2137b3d4"). InnerVolumeSpecName "kube-api-access-gn7vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:58:44.133173 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:44.133097 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn7vd\" (UniqueName: \"kubernetes.io/projected/a032506e-d4ff-49ea-a5e2-a1fa2137b3d4-kube-api-access-gn7vd\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:58:44.741392 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:44.741305 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:44.825670 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:44.825638 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9" event={"ID":"a032506e-d4ff-49ea-a5e2-a1fa2137b3d4","Type":"ContainerDied","Data":"effc21cdbd48b3e8d55cfd2663f72cee019ca3460b97b876ffc67a239ba5f05d"} Nov 03 02:58:44.825670 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:44.825660 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9" Nov 03 02:58:44.826021 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:44.825698 5463 scope.go:122] "RemoveContainer" containerID="0792f9df29ecc4a52e4e5ef1d9e5b8b6695647a8ae2b4a2afa61791ac3f3ae7a" Nov 03 02:58:44.992151 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:44.992034 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9"] Nov 03 02:58:45.042035 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:45.042002 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ecdbc7-696dd95c99-qvfg9"] Nov 03 02:58:46.444028 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:46.443995 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a032506e-d4ff-49ea-a5e2-a1fa2137b3d4" path="/var/lib/kubelet/pods/a032506e-d4ff-49ea-a5e2-a1fa2137b3d4/volumes" Nov 03 02:58:48.043621 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:48.043586 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:48.202304 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:48.202265 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:49.661664 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:49.661629 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:50.729604 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:50.729559 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:50.886329 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:50.886293 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:54.269742 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:54.269701 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:55.233480 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:55.233445 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:57.014212 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:57.013860 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:57.174790 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:57.174747 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:58.413235 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:58:58.413202 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:00.451070 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:00.451031 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:03.168701 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:03.168632 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:03.317525 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:03.317491 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:05.349620 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:05.349582 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:06.885924 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:06.885884 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:09.832211 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:09.831922 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:09.988811 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:09.988778 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:10.200348 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:10.200317 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:11.819598 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:11.819560 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:11.998173 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:11.998136 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:13.008137 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:59:13.008104 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:59:13.209087 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:13.209055 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:15.189909 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:15.189877 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:19.956275 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:19.956243 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:24.724652 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:24.724612 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:25.323073 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:25.323036 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:28.330456 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:28.330417 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:29.454602 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:29.454567 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:32.372326 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:32.372293 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9bjwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:59:32.372727 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:32.372381 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" podUID="77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:59:34.303707 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:34.303669 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:34.557255 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:34.557136 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:38.357928 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:38.357891 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt"] Nov 03 02:59:38.358471 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:38.358275 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" podUID="442cb483-89c6-4e1e-b19c-43e2741ee83d" containerName="maas-api" containerID="cri-o://66d631f01d33b4caa7855a3afd79847ef4bd32aa058a434c463a81bd82d547d8" gracePeriod=30 Nov 03 02:59:38.473419 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:38.473381 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:38.492471 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:38.492440 5463 generic.go:356] "Generic (PLEG): container finished" podID="442cb483-89c6-4e1e-b19c-43e2741ee83d" containerID="66d631f01d33b4caa7855a3afd79847ef4bd32aa058a434c463a81bd82d547d8" exitCode=0 Nov 03 02:59:38.492634 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:38.492498 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" event={"ID":"442cb483-89c6-4e1e-b19c-43e2741ee83d","Type":"ContainerDied","Data":"66d631f01d33b4caa7855a3afd79847ef4bd32aa058a434c463a81bd82d547d8"} Nov 03 02:59:38.667458 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:38.667433 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" Nov 03 02:59:38.795288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:38.792745 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/442cb483-89c6-4e1e-b19c-43e2741ee83d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/442cb483-89c6-4e1e-b19c-43e2741ee83d-metrics-certs\") pod \"442cb483-89c6-4e1e-b19c-43e2741ee83d\" (UID: \"442cb483-89c6-4e1e-b19c-43e2741ee83d\") " Nov 03 02:59:38.795288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:38.792804 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/442cb483-89c6-4e1e-b19c-43e2741ee83d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/442cb483-89c6-4e1e-b19c-43e2741ee83d-maas-api-tls\") pod \"442cb483-89c6-4e1e-b19c-43e2741ee83d\" (UID: \"442cb483-89c6-4e1e-b19c-43e2741ee83d\") " Nov 03 02:59:38.795288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:38.792921 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/442cb483-89c6-4e1e-b19c-43e2741ee83d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/442cb483-89c6-4e1e-b19c-43e2741ee83d-openshift-service-ca\") pod \"442cb483-89c6-4e1e-b19c-43e2741ee83d\" (UID: \"442cb483-89c6-4e1e-b19c-43e2741ee83d\") " Nov 03 02:59:38.795288 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:38.793017 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/442cb483-89c6-4e1e-b19c-43e2741ee83d-kube-api-access-ckblm\" (UniqueName: \"kubernetes.io/projected/442cb483-89c6-4e1e-b19c-43e2741ee83d-kube-api-access-ckblm\") pod \"442cb483-89c6-4e1e-b19c-43e2741ee83d\" (UID: \"442cb483-89c6-4e1e-b19c-43e2741ee83d\") " Nov 03 02:59:38.798334 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:38.798278 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/442cb483-89c6-4e1e-b19c-43e2741ee83d-maas-api-tls" pod "442cb483-89c6-4e1e-b19c-43e2741ee83d" (UID: "442cb483-89c6-4e1e-b19c-43e2741ee83d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:59:38.807506 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:38.807472 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/442cb483-89c6-4e1e-b19c-43e2741ee83d-openshift-service-ca" pod "442cb483-89c6-4e1e-b19c-43e2741ee83d" (UID: "442cb483-89c6-4e1e-b19c-43e2741ee83d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:59:38.810545 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:38.810499 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/442cb483-89c6-4e1e-b19c-43e2741ee83d-metrics-certs" pod "442cb483-89c6-4e1e-b19c-43e2741ee83d" (UID: "442cb483-89c6-4e1e-b19c-43e2741ee83d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:59:38.815366 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:38.815329 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442cb483-89c6-4e1e-b19c-43e2741ee83d-kube-api-access-ckblm" pod "442cb483-89c6-4e1e-b19c-43e2741ee83d" (UID: "442cb483-89c6-4e1e-b19c-43e2741ee83d"). InnerVolumeSpecName "kube-api-access-ckblm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:59:38.895598 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:38.895522 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/442cb483-89c6-4e1e-b19c-43e2741ee83d-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:38.896258 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:38.896240 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/442cb483-89c6-4e1e-b19c-43e2741ee83d-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:38.896390 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:38.896376 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/442cb483-89c6-4e1e-b19c-43e2741ee83d-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:38.896481 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:38.896467 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckblm\" (UniqueName: \"kubernetes.io/projected/442cb483-89c6-4e1e-b19c-43e2741ee83d-kube-api-access-ckblm\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:39.502726 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:39.502683 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" Nov 03 02:59:39.502726 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:39.502719 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt" event={"ID":"442cb483-89c6-4e1e-b19c-43e2741ee83d","Type":"ContainerDied","Data":"5e3d6edabf1586a8238a33b47006774a91fcaf093e7b0de9e9d49cd31ec50f26"} Nov 03 02:59:39.503299 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:39.502785 5463 scope.go:122] "RemoveContainer" containerID="66d631f01d33b4caa7855a3afd79847ef4bd32aa058a434c463a81bd82d547d8" Nov 03 02:59:39.671774 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:39.671740 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt"] Nov 03 02:59:39.721899 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:39.721867 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-95152a-dfc6fdf5b-5w7zt"] Nov 03 02:59:40.264560 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:40.264525 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:40.444127 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:40.444094 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="442cb483-89c6-4e1e-b19c-43e2741ee83d" path="/var/lib/kubelet/pods/442cb483-89c6-4e1e-b19c-43e2741ee83d/volumes" Nov 03 02:59:43.009572 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:59:43.009528 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 02:59:44.424122 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:44.424082 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:45.131979 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:45.131942 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:45.297768 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:45.297729 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:45.859973 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:45.859941 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:48.171795 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:48.171756 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:48.323687 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:48.323651 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:50.279463 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:50.279429 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:51.662289 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:51.662250 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:52.603796 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:52.603764 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6"] Nov 03 02:59:52.710767 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:52.710738 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v"] Nov 03 02:59:52.988144 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:52.988102 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:53.140869 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:53.140832 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:54.570965 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:54.570934 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:56.508033 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:56.508003 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:57.439934 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:57.439905 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:57.604218 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:57.604160 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6" podUID="dc52833c-b2ac-4cde-96e7-6b52dd86a747" containerName="payload-processing" containerID="cri-o://b143abf65a0fcb98e6bbd2703f1d26e0617e22f2b0537a5a87848196db8537e2" gracePeriod=25 Nov 03 02:59:57.711942 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:57.711867 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v" podUID="1bf418c5-21e9-4424-81a9-e928cb205a4b" containerName="payload-pre-processing" containerID="cri-o://b9077b5a7f740f7abfda16b1f1246a432bd64ab4010f19cf1fb0f0689b73e487" gracePeriod=25 Nov 03 02:59:57.930584 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:57.930558 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6" Nov 03 02:59:58.015485 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.015463 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v" Nov 03 02:59:58.082314 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.082281 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc52833c-b2ac-4cde-96e7-6b52dd86a747-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc52833c-b2ac-4cde-96e7-6b52dd86a747-plugins-config-volume\") pod \"dc52833c-b2ac-4cde-96e7-6b52dd86a747\" (UID: \"dc52833c-b2ac-4cde-96e7-6b52dd86a747\") " Nov 03 02:59:58.082512 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.082381 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc52833c-b2ac-4cde-96e7-6b52dd86a747-kube-api-access-jcx6d\" (UniqueName: \"kubernetes.io/projected/dc52833c-b2ac-4cde-96e7-6b52dd86a747-kube-api-access-jcx6d\") pod \"dc52833c-b2ac-4cde-96e7-6b52dd86a747\" (UID: \"dc52833c-b2ac-4cde-96e7-6b52dd86a747\") " Nov 03 02:59:58.082721 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.082696 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc52833c-b2ac-4cde-96e7-6b52dd86a747-plugins-config-volume" pod "dc52833c-b2ac-4cde-96e7-6b52dd86a747" (UID: "dc52833c-b2ac-4cde-96e7-6b52dd86a747"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:59:58.082985 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.082961 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc52833c-b2ac-4cde-96e7-6b52dd86a747-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:58.085005 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.084983 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc52833c-b2ac-4cde-96e7-6b52dd86a747-kube-api-access-jcx6d" pod "dc52833c-b2ac-4cde-96e7-6b52dd86a747" (UID: "dc52833c-b2ac-4cde-96e7-6b52dd86a747"). InnerVolumeSpecName "kube-api-access-jcx6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:59:58.183680 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.183637 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bf418c5-21e9-4424-81a9-e928cb205a4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf418c5-21e9-4424-81a9-e928cb205a4b-plugins-config-volume\") pod \"1bf418c5-21e9-4424-81a9-e928cb205a4b\" (UID: \"1bf418c5-21e9-4424-81a9-e928cb205a4b\") " Nov 03 02:59:58.184500 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.184479 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf418c5-21e9-4424-81a9-e928cb205a4b-kube-api-access-5pcnt\" (UniqueName: \"kubernetes.io/projected/1bf418c5-21e9-4424-81a9-e928cb205a4b-kube-api-access-5pcnt\") pod \"1bf418c5-21e9-4424-81a9-e928cb205a4b\" (UID: \"1bf418c5-21e9-4424-81a9-e928cb205a4b\") " Nov 03 02:59:58.185241 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.184275 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf418c5-21e9-4424-81a9-e928cb205a4b-plugins-config-volume" pod "1bf418c5-21e9-4424-81a9-e928cb205a4b" (UID: "1bf418c5-21e9-4424-81a9-e928cb205a4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:59:58.185504 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.185487 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf418c5-21e9-4424-81a9-e928cb205a4b-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:58.185614 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.185593 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcx6d\" (UniqueName: \"kubernetes.io/projected/dc52833c-b2ac-4cde-96e7-6b52dd86a747-kube-api-access-jcx6d\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:58.188755 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.188731 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf418c5-21e9-4424-81a9-e928cb205a4b-kube-api-access-5pcnt" pod "1bf418c5-21e9-4424-81a9-e928cb205a4b" (UID: "1bf418c5-21e9-4424-81a9-e928cb205a4b"). InnerVolumeSpecName "kube-api-access-5pcnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:59:58.287186 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.287124 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pcnt\" (UniqueName: \"kubernetes.io/projected/1bf418c5-21e9-4424-81a9-e928cb205a4b-kube-api-access-5pcnt\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:58.443803 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.443770 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:58.656623 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.656591 5463 generic.go:356] "Generic (PLEG): container finished" podID="dc52833c-b2ac-4cde-96e7-6b52dd86a747" containerID="b143abf65a0fcb98e6bbd2703f1d26e0617e22f2b0537a5a87848196db8537e2" exitCode=0 Nov 03 02:59:58.657002 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.656649 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6" Nov 03 02:59:58.657002 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.656669 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6" event={"ID":"dc52833c-b2ac-4cde-96e7-6b52dd86a747","Type":"ContainerDied","Data":"b143abf65a0fcb98e6bbd2703f1d26e0617e22f2b0537a5a87848196db8537e2"} Nov 03 02:59:58.657002 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.656735 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6" event={"ID":"dc52833c-b2ac-4cde-96e7-6b52dd86a747","Type":"ContainerDied","Data":"e2a290133e7a6f436b85c601b8f6cabe23c92436aadafe1955a9787bea303854"} Nov 03 02:59:58.657002 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.656769 5463 scope.go:122] "RemoveContainer" containerID="b143abf65a0fcb98e6bbd2703f1d26e0617e22f2b0537a5a87848196db8537e2" Nov 03 02:59:58.658306 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.658282 5463 generic.go:356] "Generic (PLEG): container finished" podID="1bf418c5-21e9-4424-81a9-e928cb205a4b" containerID="b9077b5a7f740f7abfda16b1f1246a432bd64ab4010f19cf1fb0f0689b73e487" exitCode=0 Nov 03 02:59:58.658424 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.658325 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v" Nov 03 02:59:58.658424 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.658364 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v" event={"ID":"1bf418c5-21e9-4424-81a9-e928cb205a4b","Type":"ContainerDied","Data":"b9077b5a7f740f7abfda16b1f1246a432bd64ab4010f19cf1fb0f0689b73e487"} Nov 03 02:59:58.658424 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.658409 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v" event={"ID":"1bf418c5-21e9-4424-81a9-e928cb205a4b","Type":"ContainerDied","Data":"3d04334efbb8ff5d9a3e39bf9808dd9f4c70c2c23b704da84997e5f7a089a226"} Nov 03 02:59:58.668406 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.668389 5463 scope.go:122] "RemoveContainer" containerID="b143abf65a0fcb98e6bbd2703f1d26e0617e22f2b0537a5a87848196db8537e2" Nov 03 02:59:58.668716 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:59:58.668697 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b143abf65a0fcb98e6bbd2703f1d26e0617e22f2b0537a5a87848196db8537e2\": container with ID starting with b143abf65a0fcb98e6bbd2703f1d26e0617e22f2b0537a5a87848196db8537e2 not found: ID does not exist" containerID="b143abf65a0fcb98e6bbd2703f1d26e0617e22f2b0537a5a87848196db8537e2" Nov 03 02:59:58.668819 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.668725 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b143abf65a0fcb98e6bbd2703f1d26e0617e22f2b0537a5a87848196db8537e2"} err="failed to get container status \"b143abf65a0fcb98e6bbd2703f1d26e0617e22f2b0537a5a87848196db8537e2\": rpc error: code = NotFound desc = could not find container \"b143abf65a0fcb98e6bbd2703f1d26e0617e22f2b0537a5a87848196db8537e2\": container with ID starting with b143abf65a0fcb98e6bbd2703f1d26e0617e22f2b0537a5a87848196db8537e2 not found: ID does not exist" Nov 03 02:59:58.668819 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.668748 5463 scope.go:122] "RemoveContainer" containerID="b9077b5a7f740f7abfda16b1f1246a432bd64ab4010f19cf1fb0f0689b73e487" Nov 03 02:59:58.678131 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.678114 5463 scope.go:122] "RemoveContainer" containerID="b9077b5a7f740f7abfda16b1f1246a432bd64ab4010f19cf1fb0f0689b73e487" Nov 03 02:59:58.678385 ip-10-0-137-69 kubenswrapper[5463]: E1103 02:59:58.678366 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9077b5a7f740f7abfda16b1f1246a432bd64ab4010f19cf1fb0f0689b73e487\": container with ID starting with b9077b5a7f740f7abfda16b1f1246a432bd64ab4010f19cf1fb0f0689b73e487 not found: ID does not exist" containerID="b9077b5a7f740f7abfda16b1f1246a432bd64ab4010f19cf1fb0f0689b73e487" Nov 03 02:59:58.678447 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.678391 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9077b5a7f740f7abfda16b1f1246a432bd64ab4010f19cf1fb0f0689b73e487"} err="failed to get container status \"b9077b5a7f740f7abfda16b1f1246a432bd64ab4010f19cf1fb0f0689b73e487\": rpc error: code = NotFound desc = could not find container \"b9077b5a7f740f7abfda16b1f1246a432bd64ab4010f19cf1fb0f0689b73e487\": container with ID starting with b9077b5a7f740f7abfda16b1f1246a432bd64ab4010f19cf1fb0f0689b73e487 not found: ID does not exist" Nov 03 02:59:58.715728 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.715705 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:58.824339 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.824307 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v"] Nov 03 02:59:58.875028 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.874968 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ddcbd0-db998df9f-fc77v"] Nov 03 02:59:58.938756 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.938730 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6"] Nov 03 02:59:58.943960 ip-10-0-137-69 kubenswrapper[5463]: I1103 02:59:58.943934 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ddcbd0-f4c48fcb5-w2sw6"] Nov 03 03:00:00.030220 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.030173 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:00.204606 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.204572 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv"] Nov 03 03:00:00.205338 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.205317 5463 cpu_manager.go:414] "containerMap: removing container" podUID="859202a1-dd87-427e-8e59-acfe73d4b6d8" containerName="payload-pre-processing" Nov 03 03:00:00.205474 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.205339 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="859202a1-dd87-427e-8e59-acfe73d4b6d8" containerName="payload-pre-processing" Nov 03 03:00:00.205474 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.205381 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a032506e-d4ff-49ea-a5e2-a1fa2137b3d4" containerName="payload-pre-processing" Nov 03 03:00:00.205474 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.205391 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a032506e-d4ff-49ea-a5e2-a1fa2137b3d4" containerName="payload-pre-processing" Nov 03 03:00:00.205474 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.205426 5463 cpu_manager.go:414] "containerMap: removing container" podUID="442cb483-89c6-4e1e-b19c-43e2741ee83d" containerName="maas-api" Nov 03 03:00:00.205474 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.205433 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="442cb483-89c6-4e1e-b19c-43e2741ee83d" containerName="maas-api" Nov 03 03:00:00.205474 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.205449 5463 cpu_manager.go:414] "containerMap: removing container" podUID="1bf418c5-21e9-4424-81a9-e928cb205a4b" containerName="payload-pre-processing" Nov 03 03:00:00.205474 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.205457 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bf418c5-21e9-4424-81a9-e928cb205a4b" containerName="payload-pre-processing" Nov 03 03:00:00.205849 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.205484 5463 cpu_manager.go:414] "containerMap: removing container" podUID="dc52833c-b2ac-4cde-96e7-6b52dd86a747" containerName="payload-processing" Nov 03 03:00:00.205849 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.205492 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc52833c-b2ac-4cde-96e7-6b52dd86a747" containerName="payload-processing" Nov 03 03:00:00.205849 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.205656 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bf418c5-21e9-4424-81a9-e928cb205a4b" containerName="payload-pre-processing" Nov 03 03:00:00.205849 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.205676 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc52833c-b2ac-4cde-96e7-6b52dd86a747" containerName="payload-processing" Nov 03 03:00:00.205849 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.205691 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="859202a1-dd87-427e-8e59-acfe73d4b6d8" containerName="payload-pre-processing" Nov 03 03:00:00.205849 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.205708 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a032506e-d4ff-49ea-a5e2-a1fa2137b3d4" containerName="payload-pre-processing" Nov 03 03:00:00.205849 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.205719 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="442cb483-89c6-4e1e-b19c-43e2741ee83d" containerName="maas-api" Nov 03 03:00:00.210586 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.210564 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2"] Nov 03 03:00:00.210706 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.210677 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" Nov 03 03:00:00.215326 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.215302 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh"] Nov 03 03:00:00.215474 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.215420 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" Nov 03 03:00:00.219741 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.219722 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh" Nov 03 03:00:00.256150 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.256127 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w"] Nov 03 03:00:00.261403 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.261383 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n"] Nov 03 03:00:00.261507 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.261488 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" Nov 03 03:00:00.266578 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.266561 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6"] Nov 03 03:00:00.266705 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.266685 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" Nov 03 03:00:00.271302 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.271285 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" Nov 03 03:00:00.329372 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.329339 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pgbx4\"" Nov 03 03:00:00.337367 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.337330 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpww8\" (UniqueName: \"kubernetes.io/projected/d7ddd193-7cb2-401b-93b9-407faf6ed1fc-kube-api-access-bpww8\") pod \"maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2\" (UID: \"d7ddd193-7cb2-401b-93b9-407faf6ed1fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" Nov 03 03:00:00.337524 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.337426 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e57752-f49f-4fda-a9f7-8f902dfd6fb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh\" (UID: \"a6e57752-f49f-4fda-a9f7-8f902dfd6fb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh" Nov 03 03:00:00.337587 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.337526 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w8rd\" (UniqueName: \"kubernetes.io/projected/f1300f44-ac64-49c0-9387-8279a643f646-kube-api-access-6w8rd\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv\" (UID: \"f1300f44-ac64-49c0-9387-8279a643f646\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" Nov 03 03:00:00.337587 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.337564 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1300f44-ac64-49c0-9387-8279a643f646-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv\" (UID: \"f1300f44-ac64-49c0-9387-8279a643f646\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" Nov 03 03:00:00.337664 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.337589 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ddd193-7cb2-401b-93b9-407faf6ed1fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2\" (UID: \"d7ddd193-7cb2-401b-93b9-407faf6ed1fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" Nov 03 03:00:00.337664 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.337637 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxlf2\" (UniqueName: \"kubernetes.io/projected/a6e57752-f49f-4fda-a9f7-8f902dfd6fb2-kube-api-access-wxlf2\") pod \"maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh\" (UID: \"a6e57752-f49f-4fda-a9f7-8f902dfd6fb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh" Nov 03 03:00:00.340043 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.340020 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv"] Nov 03 03:00:00.353236 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.353027 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2"] Nov 03 03:00:00.387944 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.387915 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh"] Nov 03 03:00:00.393670 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.393638 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n"] Nov 03 03:00:00.398605 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.398575 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6"] Nov 03 03:00:00.413498 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.413470 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w"] Nov 03 03:00:00.438751 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.438720 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/510e94f4-1d02-4c4b-8928-6253787c69e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894580-h8t6n\" (UID: \"510e94f4-1d02-4c4b-8928-6253787c69e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" Nov 03 03:00:00.438927 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.438765 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae465e23-7552-4847-8526-028628c3cb05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w\" (UID: \"ae465e23-7552-4847-8526-028628c3cb05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" Nov 03 03:00:00.438927 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.438879 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sscbx\" (UniqueName: \"kubernetes.io/projected/0903eaf9-cfca-4e03-8200-561cff5ab89f-kube-api-access-sscbx\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6\" (UID: \"0903eaf9-cfca-4e03-8200-561cff5ab89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" Nov 03 03:00:00.438927 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.438920 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0903eaf9-cfca-4e03-8200-561cff5ab89f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6\" (UID: \"0903eaf9-cfca-4e03-8200-561cff5ab89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" Nov 03 03:00:00.439088 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.438986 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9n4t\" (UniqueName: \"kubernetes.io/projected/510e94f4-1d02-4c4b-8928-6253787c69e4-kube-api-access-m9n4t\") pod \"maas-api-key-cleanup-29894580-h8t6n\" (UID: \"510e94f4-1d02-4c4b-8928-6253787c69e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" Nov 03 03:00:00.439088 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.439034 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpww8\" (UniqueName: \"kubernetes.io/projected/d7ddd193-7cb2-401b-93b9-407faf6ed1fc-kube-api-access-bpww8\") pod \"maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2\" (UID: \"d7ddd193-7cb2-401b-93b9-407faf6ed1fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" Nov 03 03:00:00.439088 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.439072 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e57752-f49f-4fda-a9f7-8f902dfd6fb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh\" (UID: \"a6e57752-f49f-4fda-a9f7-8f902dfd6fb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh" Nov 03 03:00:00.439637 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.439596 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w8rd\" (UniqueName: \"kubernetes.io/projected/f1300f44-ac64-49c0-9387-8279a643f646-kube-api-access-6w8rd\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv\" (UID: \"f1300f44-ac64-49c0-9387-8279a643f646\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" Nov 03 03:00:00.439736 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.439655 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1300f44-ac64-49c0-9387-8279a643f646-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv\" (UID: \"f1300f44-ac64-49c0-9387-8279a643f646\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" Nov 03 03:00:00.439736 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.439682 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ddd193-7cb2-401b-93b9-407faf6ed1fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2\" (UID: \"d7ddd193-7cb2-401b-93b9-407faf6ed1fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" Nov 03 03:00:00.439736 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.439727 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxlf2\" (UniqueName: \"kubernetes.io/projected/a6e57752-f49f-4fda-a9f7-8f902dfd6fb2-kube-api-access-wxlf2\") pod \"maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh\" (UID: \"a6e57752-f49f-4fda-a9f7-8f902dfd6fb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh" Nov 03 03:00:00.439913 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.439781 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc4lk\" (UniqueName: \"kubernetes.io/projected/ae465e23-7552-4847-8526-028628c3cb05-kube-api-access-vc4lk\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w\" (UID: \"ae465e23-7552-4847-8526-028628c3cb05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" Nov 03 03:00:00.441317 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.440909 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e57752-f49f-4fda-a9f7-8f902dfd6fb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh\" (UID: \"a6e57752-f49f-4fda-a9f7-8f902dfd6fb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh" Nov 03 03:00:00.441317 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.440951 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1300f44-ac64-49c0-9387-8279a643f646-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv\" (UID: \"f1300f44-ac64-49c0-9387-8279a643f646\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" Nov 03 03:00:00.441539 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.441438 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ddd193-7cb2-401b-93b9-407faf6ed1fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2\" (UID: \"d7ddd193-7cb2-401b-93b9-407faf6ed1fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" Nov 03 03:00:00.448030 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.447483 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bf418c5-21e9-4424-81a9-e928cb205a4b" path="/var/lib/kubelet/pods/1bf418c5-21e9-4424-81a9-e928cb205a4b/volumes" Nov 03 03:00:00.448148 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.448062 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc52833c-b2ac-4cde-96e7-6b52dd86a747" path="/var/lib/kubelet/pods/dc52833c-b2ac-4cde-96e7-6b52dd86a747/volumes" Nov 03 03:00:00.484646 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.484612 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh"] Nov 03 03:00:00.484959 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.484890 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" podUID="bc8f16d1-4d9c-4c4b-8449-689172ef80be" containerName="istio-proxy" containerID="cri-o://20a96e641da37d8465242c1bb60de97266b7ec6ef9f0d73d2196bdaf659212e0" gracePeriod=30 Nov 03 03:00:00.502001 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.501922 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpww8\" (UniqueName: \"kubernetes.io/projected/d7ddd193-7cb2-401b-93b9-407faf6ed1fc-kube-api-access-bpww8\") pod \"maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2\" (UID: \"d7ddd193-7cb2-401b-93b9-407faf6ed1fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" Nov 03 03:00:00.502388 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.502367 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w8rd\" (UniqueName: \"kubernetes.io/projected/f1300f44-ac64-49c0-9387-8279a643f646-kube-api-access-6w8rd\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv\" (UID: \"f1300f44-ac64-49c0-9387-8279a643f646\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" Nov 03 03:00:00.503090 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.503062 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxlf2\" (UniqueName: \"kubernetes.io/projected/a6e57752-f49f-4fda-a9f7-8f902dfd6fb2-kube-api-access-wxlf2\") pod \"maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh\" (UID: \"a6e57752-f49f-4fda-a9f7-8f902dfd6fb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh" Nov 03 03:00:00.526281 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.526258 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" Nov 03 03:00:00.534861 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.534842 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" Nov 03 03:00:00.540762 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.540734 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sscbx\" (UniqueName: \"kubernetes.io/projected/0903eaf9-cfca-4e03-8200-561cff5ab89f-kube-api-access-sscbx\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6\" (UID: \"0903eaf9-cfca-4e03-8200-561cff5ab89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" Nov 03 03:00:00.540885 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.540801 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0903eaf9-cfca-4e03-8200-561cff5ab89f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6\" (UID: \"0903eaf9-cfca-4e03-8200-561cff5ab89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" Nov 03 03:00:00.540885 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.540839 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9n4t\" (UniqueName: \"kubernetes.io/projected/510e94f4-1d02-4c4b-8928-6253787c69e4-kube-api-access-m9n4t\") pod \"maas-api-key-cleanup-29894580-h8t6n\" (UID: \"510e94f4-1d02-4c4b-8928-6253787c69e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" Nov 03 03:00:00.540885 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.540845 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh" Nov 03 03:00:00.541071 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.541043 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:00.541141 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.541094 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc4lk\" (UniqueName: \"kubernetes.io/projected/ae465e23-7552-4847-8526-028628c3cb05-kube-api-access-vc4lk\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w\" (UID: \"ae465e23-7552-4847-8526-028628c3cb05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" Nov 03 03:00:00.542116 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.541173 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/510e94f4-1d02-4c4b-8928-6253787c69e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894580-h8t6n\" (UID: \"510e94f4-1d02-4c4b-8928-6253787c69e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" Nov 03 03:00:00.542116 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.542087 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae465e23-7552-4847-8526-028628c3cb05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w\" (UID: \"ae465e23-7552-4847-8526-028628c3cb05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" Nov 03 03:00:00.542590 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.542566 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0903eaf9-cfca-4e03-8200-561cff5ab89f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6\" (UID: \"0903eaf9-cfca-4e03-8200-561cff5ab89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" Nov 03 03:00:00.544822 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.544795 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae465e23-7552-4847-8526-028628c3cb05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w\" (UID: \"ae465e23-7552-4847-8526-028628c3cb05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" Nov 03 03:00:00.545483 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.545459 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/510e94f4-1d02-4c4b-8928-6253787c69e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894580-h8t6n\" (UID: \"510e94f4-1d02-4c4b-8928-6253787c69e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" Nov 03 03:00:00.649268 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.642693 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sscbx\" (UniqueName: \"kubernetes.io/projected/0903eaf9-cfca-4e03-8200-561cff5ab89f-kube-api-access-sscbx\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6\" (UID: \"0903eaf9-cfca-4e03-8200-561cff5ab89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" Nov 03 03:00:00.649268 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.642882 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9n4t\" (UniqueName: \"kubernetes.io/projected/510e94f4-1d02-4c4b-8928-6253787c69e4-kube-api-access-m9n4t\") pod \"maas-api-key-cleanup-29894580-h8t6n\" (UID: \"510e94f4-1d02-4c4b-8928-6253787c69e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" Nov 03 03:00:00.678217 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.664881 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc4lk\" (UniqueName: \"kubernetes.io/projected/ae465e23-7552-4847-8526-028628c3cb05-kube-api-access-vc4lk\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w\" (UID: \"ae465e23-7552-4847-8526-028628c3cb05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" Nov 03 03:00:00.877170 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.877133 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" Nov 03 03:00:00.887615 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.887525 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" Nov 03 03:00:00.895514 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:00.895343 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" Nov 03 03:00:01.080466 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:01.080433 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv"] Nov 03 03:00:01.115842 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:01.115809 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2"] Nov 03 03:00:01.148027 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:00:01.147998 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e57752_f49f_4fda_a9f7_8f902dfd6fb2.slice/crio-ac258face9382ce558aa0fcfb9b787ab6afcf948b175d34afdf4e54ec168761a WatchSource:0}: Error finding container ac258face9382ce558aa0fcfb9b787ab6afcf948b175d34afdf4e54ec168761a: Status 404 returned error can't find the container with id ac258face9382ce558aa0fcfb9b787ab6afcf948b175d34afdf4e54ec168761a Nov 03 03:00:01.152848 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:01.152820 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh"] Nov 03 03:00:01.316439 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:00:01.316413 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0903eaf9_cfca_4e03_8200_561cff5ab89f.slice/crio-0c53dafded6a43acb9174bd85744c868d0a7f2f632cfa3bfc5459c128a710a28 WatchSource:0}: Error finding container 0c53dafded6a43acb9174bd85744c868d0a7f2f632cfa3bfc5459c128a710a28: Status 404 returned error can't find the container with id 0c53dafded6a43acb9174bd85744c868d0a7f2f632cfa3bfc5459c128a710a28 Nov 03 03:00:01.319967 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:01.319944 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6"] Nov 03 03:00:01.354871 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:00:01.354846 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510e94f4_1d02_4c4b_8928_6253787c69e4.slice/crio-ce35ba4c5a97d0a5a1ce8740e4cfbd3e3a57f1fa15e114bcfbe3fa2221e7a5e3 WatchSource:0}: Error finding container ce35ba4c5a97d0a5a1ce8740e4cfbd3e3a57f1fa15e114bcfbe3fa2221e7a5e3: Status 404 returned error can't find the container with id ce35ba4c5a97d0a5a1ce8740e4cfbd3e3a57f1fa15e114bcfbe3fa2221e7a5e3 Nov 03 03:00:01.355621 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:01.355525 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n"] Nov 03 03:00:01.359425 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:01.359400 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w"] Nov 03 03:00:01.387869 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:01.387843 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:01.451884 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:00:01.451840 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae465e23_7552_4847_8526_028628c3cb05.slice/crio-d5a22c16ceb59a99a263720272e398f0545836bed9040fa94af2ee358981f3ae WatchSource:0}: Error finding container d5a22c16ceb59a99a263720272e398f0545836bed9040fa94af2ee358981f3ae: Status 404 returned error can't find the container with id d5a22c16ceb59a99a263720272e398f0545836bed9040fa94af2ee358981f3ae Nov 03 03:00:01.594325 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:01.594224 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:01.701629 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:01.701596 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh" event={"ID":"a6e57752-f49f-4fda-a9f7-8f902dfd6fb2","Type":"ContainerStarted","Data":"ac258face9382ce558aa0fcfb9b787ab6afcf948b175d34afdf4e54ec168761a"} Nov 03 03:00:01.703458 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:01.703435 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" event={"ID":"d7ddd193-7cb2-401b-93b9-407faf6ed1fc","Type":"ContainerStarted","Data":"8a5e124633557548c6632646b1cba6682e16908345ccf8b029f71e687ed93406"} Nov 03 03:00:01.703571 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:01.703479 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" event={"ID":"d7ddd193-7cb2-401b-93b9-407faf6ed1fc","Type":"ContainerStarted","Data":"859c55bf7d0858d88881fa674321f40e797d464ac7c117f707280b04b9aa4299"} Nov 03 03:00:01.704844 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:01.704823 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" event={"ID":"0903eaf9-cfca-4e03-8200-561cff5ab89f","Type":"ContainerStarted","Data":"0c53dafded6a43acb9174bd85744c868d0a7f2f632cfa3bfc5459c128a710a28"} Nov 03 03:00:01.706047 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:01.706021 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" event={"ID":"510e94f4-1d02-4c4b-8928-6253787c69e4","Type":"ContainerStarted","Data":"ce35ba4c5a97d0a5a1ce8740e4cfbd3e3a57f1fa15e114bcfbe3fa2221e7a5e3"} Nov 03 03:00:01.707425 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:01.707398 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" event={"ID":"ae465e23-7552-4847-8526-028628c3cb05","Type":"ContainerStarted","Data":"d5a22c16ceb59a99a263720272e398f0545836bed9040fa94af2ee358981f3ae"} Nov 03 03:00:01.709011 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:01.708976 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" event={"ID":"f1300f44-ac64-49c0-9387-8279a643f646","Type":"ContainerStarted","Data":"2100127ca51470c6f1c03ee5158fe29a8f53721ccddbe218538c05d883430655"} Nov 03 03:00:01.709112 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:01.709016 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" event={"ID":"f1300f44-ac64-49c0-9387-8279a643f646","Type":"ContainerStarted","Data":"ab53fb2319e221aacc6ab4bb3a62931a250e90b8f453e2f6482fca891e552ceb"} Nov 03 03:00:01.820711 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:01.820661 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" podStartSLOduration=1.820646802 podStartE2EDuration="1.820646802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:01.816896355 +0000 UTC m=+4083.501005511" watchObservedRunningTime="2026-11-03 03:00:01.820646802 +0000 UTC m=+4083.504755995" Nov 03 03:00:01.933391 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:01.933337 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" podStartSLOduration=1.933321457 podStartE2EDuration="1.933321457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:01.929935946 +0000 UTC m=+4083.614045101" watchObservedRunningTime="2026-11-03 03:00:01.933321457 +0000 UTC m=+4083.617430612" Nov 03 03:00:02.718170 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:02.718133 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" event={"ID":"ae465e23-7552-4847-8526-028628c3cb05","Type":"ContainerStarted","Data":"a10b08bb1fd34f35d4620391f7ee1b55962c2082b8a991733bf58d2d0e43dd5c"} Nov 03 03:00:02.719913 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:02.719888 5463 generic.go:356] "Generic (PLEG): container finished" podID="f1300f44-ac64-49c0-9387-8279a643f646" containerID="2100127ca51470c6f1c03ee5158fe29a8f53721ccddbe218538c05d883430655" exitCode=60 Nov 03 03:00:02.720049 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:02.719972 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" event={"ID":"f1300f44-ac64-49c0-9387-8279a643f646","Type":"ContainerDied","Data":"2100127ca51470c6f1c03ee5158fe29a8f53721ccddbe218538c05d883430655"} Nov 03 03:00:02.720236 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:02.720220 5463 scope.go:122] "RemoveContainer" containerID="2100127ca51470c6f1c03ee5158fe29a8f53721ccddbe218538c05d883430655" Nov 03 03:00:02.721742 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:02.721720 5463 generic.go:356] "Generic (PLEG): container finished" podID="a6e57752-f49f-4fda-a9f7-8f902dfd6fb2" containerID="a244e1273ebe87a5e8c2b56fd7421e7c1c306cb94d3213234e73ffeeac116748" exitCode=60 Nov 03 03:00:02.721828 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:02.721748 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh" event={"ID":"a6e57752-f49f-4fda-a9f7-8f902dfd6fb2","Type":"ContainerDied","Data":"a244e1273ebe87a5e8c2b56fd7421e7c1c306cb94d3213234e73ffeeac116748"} Nov 03 03:00:02.722058 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:02.722044 5463 scope.go:122] "RemoveContainer" containerID="a244e1273ebe87a5e8c2b56fd7421e7c1c306cb94d3213234e73ffeeac116748" Nov 03 03:00:02.723564 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:02.723450 5463 generic.go:356] "Generic (PLEG): container finished" podID="d7ddd193-7cb2-401b-93b9-407faf6ed1fc" containerID="8a5e124633557548c6632646b1cba6682e16908345ccf8b029f71e687ed93406" exitCode=60 Nov 03 03:00:02.723564 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:02.723536 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" event={"ID":"d7ddd193-7cb2-401b-93b9-407faf6ed1fc","Type":"ContainerDied","Data":"8a5e124633557548c6632646b1cba6682e16908345ccf8b029f71e687ed93406"} Nov 03 03:00:02.723802 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:02.723694 5463 scope.go:122] "RemoveContainer" containerID="8a5e124633557548c6632646b1cba6682e16908345ccf8b029f71e687ed93406" Nov 03 03:00:02.725144 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:02.725125 5463 generic.go:356] "Generic (PLEG): container finished" podID="0903eaf9-cfca-4e03-8200-561cff5ab89f" containerID="27169f08fd681ee9c2bf1d1194c35f92e3f79bf68358aff63b0b26bc80afaf3a" exitCode=60 Nov 03 03:00:02.725366 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:02.725232 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" event={"ID":"0903eaf9-cfca-4e03-8200-561cff5ab89f","Type":"ContainerDied","Data":"27169f08fd681ee9c2bf1d1194c35f92e3f79bf68358aff63b0b26bc80afaf3a"} Nov 03 03:00:02.725422 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:02.725382 5463 scope.go:122] "RemoveContainer" containerID="27169f08fd681ee9c2bf1d1194c35f92e3f79bf68358aff63b0b26bc80afaf3a" Nov 03 03:00:02.726841 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:02.726822 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" event={"ID":"510e94f4-1d02-4c4b-8928-6253787c69e4","Type":"ContainerStarted","Data":"98c7575da0036ac88ee41cacefc8efa7049f294600e7e379442f894212557078"} Nov 03 03:00:02.833427 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:02.833377 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" podStartSLOduration=2.833363599 podStartE2EDuration="2.833363599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:02.831709564 +0000 UTC m=+4084.515818719" watchObservedRunningTime="2026-11-03 03:00:02.833363599 +0000 UTC m=+4084.517472808" Nov 03 03:00:03.402640 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:03.402597 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" podStartSLOduration=3.402583183 podStartE2EDuration="3.402583183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:03.401949474 +0000 UTC m=+4085.086058628" watchObservedRunningTime="2026-11-03 03:00:03.402583183 +0000 UTC m=+4085.086692337" Nov 03 03:00:03.736152 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:03.736110 5463 generic.go:356] "Generic (PLEG): container finished" podID="ae465e23-7552-4847-8526-028628c3cb05" containerID="a10b08bb1fd34f35d4620391f7ee1b55962c2082b8a991733bf58d2d0e43dd5c" exitCode=60 Nov 03 03:00:03.736589 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:03.736228 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" event={"ID":"ae465e23-7552-4847-8526-028628c3cb05","Type":"ContainerDied","Data":"a10b08bb1fd34f35d4620391f7ee1b55962c2082b8a991733bf58d2d0e43dd5c"} Nov 03 03:00:03.736589 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:03.736479 5463 scope.go:122] "RemoveContainer" containerID="a10b08bb1fd34f35d4620391f7ee1b55962c2082b8a991733bf58d2d0e43dd5c" Nov 03 03:00:03.738330 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:03.738307 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" event={"ID":"f1300f44-ac64-49c0-9387-8279a643f646","Type":"ContainerStarted","Data":"8e04def2f1cc5292821db30d8df8ff164008c2f55fba671e18ed697c56f765e3"} Nov 03 03:00:03.740532 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:03.740510 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh" event={"ID":"a6e57752-f49f-4fda-a9f7-8f902dfd6fb2","Type":"ContainerStarted","Data":"3e1f936b53c2fab544ab871ae962a76e6b52ff0b8d48d401cfbc8d545f93f325"} Nov 03 03:00:03.742518 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:03.742495 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" event={"ID":"d7ddd193-7cb2-401b-93b9-407faf6ed1fc","Type":"ContainerStarted","Data":"2ff4c616b7498af37cbda60a3c1d69dfd09e15f285ad93d3441516c624e106fb"} Nov 03 03:00:03.744443 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:03.744420 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" event={"ID":"0903eaf9-cfca-4e03-8200-561cff5ab89f","Type":"ContainerStarted","Data":"2577b03ef2e17ad23463cbe20ea6fe5307c7d1e88d254c3ae6011241644e6bc6"} Nov 03 03:00:03.745853 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:03.745832 5463 generic.go:356] "Generic (PLEG): container finished" podID="510e94f4-1d02-4c4b-8928-6253787c69e4" containerID="98c7575da0036ac88ee41cacefc8efa7049f294600e7e379442f894212557078" exitCode=60 Nov 03 03:00:03.745951 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:03.745881 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" event={"ID":"510e94f4-1d02-4c4b-8928-6253787c69e4","Type":"ContainerDied","Data":"98c7575da0036ac88ee41cacefc8efa7049f294600e7e379442f894212557078"} Nov 03 03:00:03.746068 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:03.746053 5463 scope.go:122] "RemoveContainer" containerID="98c7575da0036ac88ee41cacefc8efa7049f294600e7e379442f894212557078" Nov 03 03:00:03.968433 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:03.968375 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" podStartSLOduration=3.96835383 podStartE2EDuration="3.96835383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:03.965280514 +0000 UTC m=+4085.649389670" watchObservedRunningTime="2026-11-03 03:00:03.96835383 +0000 UTC m=+4085.652462987" Nov 03 03:00:04.425494 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:04.425446 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh" podStartSLOduration=4.425432933 podStartE2EDuration="4.425432933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:04.424506147 +0000 UTC m=+4086.108615301" watchObservedRunningTime="2026-11-03 03:00:04.425432933 +0000 UTC m=+4086.109542087" Nov 03 03:00:04.756785 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:04.756749 5463 generic.go:356] "Generic (PLEG): container finished" podID="a6e57752-f49f-4fda-a9f7-8f902dfd6fb2" containerID="3e1f936b53c2fab544ab871ae962a76e6b52ff0b8d48d401cfbc8d545f93f325" exitCode=60 Nov 03 03:00:04.757291 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:04.756837 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh" event={"ID":"a6e57752-f49f-4fda-a9f7-8f902dfd6fb2","Type":"ContainerDied","Data":"3e1f936b53c2fab544ab871ae962a76e6b52ff0b8d48d401cfbc8d545f93f325"} Nov 03 03:00:04.757291 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:04.756908 5463 scope.go:122] "RemoveContainer" containerID="a244e1273ebe87a5e8c2b56fd7421e7c1c306cb94d3213234e73ffeeac116748" Nov 03 03:00:04.757291 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:04.757134 5463 scope.go:122] "RemoveContainer" containerID="3e1f936b53c2fab544ab871ae962a76e6b52ff0b8d48d401cfbc8d545f93f325" Nov 03 03:00:04.757564 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:04.757525 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh_odh-ai-gateway-infra(a6e57752-f49f-4fda-a9f7-8f902dfd6fb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh" podUID="a6e57752-f49f-4fda-a9f7-8f902dfd6fb2" Nov 03 03:00:04.760068 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:04.760045 5463 generic.go:356] "Generic (PLEG): container finished" podID="d7ddd193-7cb2-401b-93b9-407faf6ed1fc" containerID="2ff4c616b7498af37cbda60a3c1d69dfd09e15f285ad93d3441516c624e106fb" exitCode=60 Nov 03 03:00:04.760204 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:04.760167 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" event={"ID":"d7ddd193-7cb2-401b-93b9-407faf6ed1fc","Type":"ContainerDied","Data":"2ff4c616b7498af37cbda60a3c1d69dfd09e15f285ad93d3441516c624e106fb"} Nov 03 03:00:04.760430 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:04.760413 5463 scope.go:122] "RemoveContainer" containerID="2ff4c616b7498af37cbda60a3c1d69dfd09e15f285ad93d3441516c624e106fb" Nov 03 03:00:04.760731 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:04.760710 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2_odh-ai-gateway-infra(d7ddd193-7cb2-401b-93b9-407faf6ed1fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" podUID="d7ddd193-7cb2-401b-93b9-407faf6ed1fc" Nov 03 03:00:04.763102 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:04.763078 5463 generic.go:356] "Generic (PLEG): container finished" podID="0903eaf9-cfca-4e03-8200-561cff5ab89f" containerID="2577b03ef2e17ad23463cbe20ea6fe5307c7d1e88d254c3ae6011241644e6bc6" exitCode=60 Nov 03 03:00:04.763254 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:04.763160 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" event={"ID":"0903eaf9-cfca-4e03-8200-561cff5ab89f","Type":"ContainerDied","Data":"2577b03ef2e17ad23463cbe20ea6fe5307c7d1e88d254c3ae6011241644e6bc6"} Nov 03 03:00:04.763829 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:04.763735 5463 scope.go:122] "RemoveContainer" containerID="2577b03ef2e17ad23463cbe20ea6fe5307c7d1e88d254c3ae6011241644e6bc6" Nov 03 03:00:04.764046 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:04.764020 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6_odh-ai-gateway-infra(0903eaf9-cfca-4e03-8200-561cff5ab89f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" podUID="0903eaf9-cfca-4e03-8200-561cff5ab89f" Nov 03 03:00:04.765685 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:04.765662 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" event={"ID":"510e94f4-1d02-4c4b-8928-6253787c69e4","Type":"ContainerStarted","Data":"2081a884d3e1f00ab9a9113842a6489a5b5f56b3eaa28205af58a5d590451afa"} Nov 03 03:00:04.767723 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:04.767701 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" event={"ID":"ae465e23-7552-4847-8526-028628c3cb05","Type":"ContainerStarted","Data":"3f09a62605a10ded604aafea78f64a3d635ee372ce21c9198e89c11c85dfafdc"} Nov 03 03:00:04.769704 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:04.769679 5463 generic.go:356] "Generic (PLEG): container finished" podID="f1300f44-ac64-49c0-9387-8279a643f646" containerID="8e04def2f1cc5292821db30d8df8ff164008c2f55fba671e18ed697c56f765e3" exitCode=60 Nov 03 03:00:04.769801 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:04.769763 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" event={"ID":"f1300f44-ac64-49c0-9387-8279a643f646","Type":"ContainerDied","Data":"8e04def2f1cc5292821db30d8df8ff164008c2f55fba671e18ed697c56f765e3"} Nov 03 03:00:04.769924 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:04.769910 5463 scope.go:122] "RemoveContainer" containerID="8e04def2f1cc5292821db30d8df8ff164008c2f55fba671e18ed697c56f765e3" Nov 03 03:00:04.770134 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:04.770117 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv_odh-ai-gateway-infra(f1300f44-ac64-49c0-9387-8279a643f646)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" podUID="f1300f44-ac64-49c0-9387-8279a643f646" Nov 03 03:00:04.781662 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:04.781648 5463 scope.go:122] "RemoveContainer" containerID="8a5e124633557548c6632646b1cba6682e16908345ccf8b029f71e687ed93406" Nov 03 03:00:04.797725 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:04.797704 5463 scope.go:122] "RemoveContainer" containerID="27169f08fd681ee9c2bf1d1194c35f92e3f79bf68358aff63b0b26bc80afaf3a" Nov 03 03:00:04.811227 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:04.811178 5463 scope.go:122] "RemoveContainer" containerID="2100127ca51470c6f1c03ee5158fe29a8f53721ccddbe218538c05d883430655" Nov 03 03:00:05.780221 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.780167 5463 generic.go:356] "Generic (PLEG): container finished" podID="510e94f4-1d02-4c4b-8928-6253787c69e4" containerID="2081a884d3e1f00ab9a9113842a6489a5b5f56b3eaa28205af58a5d590451afa" exitCode=60 Nov 03 03:00:05.780574 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.780253 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" event={"ID":"510e94f4-1d02-4c4b-8928-6253787c69e4","Type":"ContainerDied","Data":"2081a884d3e1f00ab9a9113842a6489a5b5f56b3eaa28205af58a5d590451afa"} Nov 03 03:00:05.780574 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.780316 5463 scope.go:122] "RemoveContainer" containerID="98c7575da0036ac88ee41cacefc8efa7049f294600e7e379442f894212557078" Nov 03 03:00:05.780669 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.780634 5463 scope.go:122] "RemoveContainer" containerID="2081a884d3e1f00ab9a9113842a6489a5b5f56b3eaa28205af58a5d590451afa" Nov 03 03:00:05.781033 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:05.781011 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894580-h8t6n_odh-ai-gateway-infra(510e94f4-1d02-4c4b-8928-6253787c69e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" podUID="510e94f4-1d02-4c4b-8928-6253787c69e4" Nov 03 03:00:05.783113 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.783088 5463 generic.go:356] "Generic (PLEG): container finished" podID="ae465e23-7552-4847-8526-028628c3cb05" containerID="3f09a62605a10ded604aafea78f64a3d635ee372ce21c9198e89c11c85dfafdc" exitCode=60 Nov 03 03:00:05.783259 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.783168 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" event={"ID":"ae465e23-7552-4847-8526-028628c3cb05","Type":"ContainerDied","Data":"3f09a62605a10ded604aafea78f64a3d635ee372ce21c9198e89c11c85dfafdc"} Nov 03 03:00:05.783458 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.783442 5463 scope.go:122] "RemoveContainer" containerID="3f09a62605a10ded604aafea78f64a3d635ee372ce21c9198e89c11c85dfafdc" Nov 03 03:00:05.783786 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:05.783765 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w_odh-ai-gateway-infra(ae465e23-7552-4847-8526-028628c3cb05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" podUID="ae465e23-7552-4847-8526-028628c3cb05" Nov 03 03:00:05.785306 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.785282 5463 scope.go:122] "RemoveContainer" containerID="8e04def2f1cc5292821db30d8df8ff164008c2f55fba671e18ed697c56f765e3" Nov 03 03:00:05.785626 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:05.785603 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv_odh-ai-gateway-infra(f1300f44-ac64-49c0-9387-8279a643f646)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" podUID="f1300f44-ac64-49c0-9387-8279a643f646" Nov 03 03:00:05.786888 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.786855 5463 generic.go:356] "Generic (PLEG): container finished" podID="bc8f16d1-4d9c-4c4b-8449-689172ef80be" containerID="20a96e641da37d8465242c1bb60de97266b7ec6ef9f0d73d2196bdaf659212e0" exitCode=0 Nov 03 03:00:05.787002 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.786926 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" event={"ID":"bc8f16d1-4d9c-4c4b-8449-689172ef80be","Type":"ContainerDied","Data":"20a96e641da37d8465242c1bb60de97266b7ec6ef9f0d73d2196bdaf659212e0"} Nov 03 03:00:05.787002 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.786988 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" event={"ID":"bc8f16d1-4d9c-4c4b-8449-689172ef80be","Type":"ContainerDied","Data":"463c6a3b12058a121d266ba2f018e30748b4d3ad867227ddb3ed1e8877f51a95"} Nov 03 03:00:05.787099 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.787015 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="463c6a3b12058a121d266ba2f018e30748b4d3ad867227ddb3ed1e8877f51a95" Nov 03 03:00:05.788973 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.788956 5463 scope.go:122] "RemoveContainer" containerID="3e1f936b53c2fab544ab871ae962a76e6b52ff0b8d48d401cfbc8d545f93f325" Nov 03 03:00:05.789219 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:05.789184 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh_odh-ai-gateway-infra(a6e57752-f49f-4fda-a9f7-8f902dfd6fb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh" podUID="a6e57752-f49f-4fda-a9f7-8f902dfd6fb2" Nov 03 03:00:05.791141 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.791121 5463 scope.go:122] "RemoveContainer" containerID="2ff4c616b7498af37cbda60a3c1d69dfd09e15f285ad93d3441516c624e106fb" Nov 03 03:00:05.791445 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:05.791421 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2_odh-ai-gateway-infra(d7ddd193-7cb2-401b-93b9-407faf6ed1fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" podUID="d7ddd193-7cb2-401b-93b9-407faf6ed1fc" Nov 03 03:00:05.793497 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.793476 5463 scope.go:122] "RemoveContainer" containerID="2577b03ef2e17ad23463cbe20ea6fe5307c7d1e88d254c3ae6011241644e6bc6" Nov 03 03:00:05.793784 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:05.793747 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6_odh-ai-gateway-infra(0903eaf9-cfca-4e03-8200-561cff5ab89f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" podUID="0903eaf9-cfca-4e03-8200-561cff5ab89f" Nov 03 03:00:05.810643 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.810628 5463 scope.go:122] "RemoveContainer" containerID="a10b08bb1fd34f35d4620391f7ee1b55962c2082b8a991733bf58d2d0e43dd5c" Nov 03 03:00:05.812516 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.812501 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 03:00:05.822997 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.822972 5463 scope.go:122] "RemoveContainer" containerID="12cf0db4a1a4f1e00c579a1aaa3e4b354df4638f8b28dd474a8915bdbf9fbff9" Nov 03 03:00:05.841222 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.841201 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc8f16d1-4d9c-4c4b-8449-689172ef80be-kube-api-access-spc47\" (UniqueName: \"kubernetes.io/projected/bc8f16d1-4d9c-4c4b-8449-689172ef80be-kube-api-access-spc47\") pod \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " Nov 03 03:00:05.841325 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.841248 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-data\") pod \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " Nov 03 03:00:05.841325 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.841287 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-workload-socket\") pod \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " Nov 03 03:00:05.841408 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.841351 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-credential-socket\") pod \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " Nov 03 03:00:05.841408 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.841393 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-podinfo\") pod \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " Nov 03 03:00:05.841524 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.841419 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-workload-certs\") pod \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " Nov 03 03:00:05.841524 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.841442 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-envoy\") pod \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " Nov 03 03:00:05.841524 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.841441 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-data" pod "bc8f16d1-4d9c-4c4b-8449-689172ef80be" (UID: "bc8f16d1-4d9c-4c4b-8449-689172ef80be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:05.841524 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.841483 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-token\" (UniqueName: \"kubernetes.io/projected/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-token\") pod \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " Nov 03 03:00:05.841735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.841541 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istiod-ca-cert\") pod \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\" (UID: \"bc8f16d1-4d9c-4c4b-8449-689172ef80be\") " Nov 03 03:00:05.841735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.841572 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-workload-socket" pod "bc8f16d1-4d9c-4c4b-8449-689172ef80be" (UID: "bc8f16d1-4d9c-4c4b-8449-689172ef80be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:05.841735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.841641 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-workload-certs" pod "bc8f16d1-4d9c-4c4b-8449-689172ef80be" (UID: "bc8f16d1-4d9c-4c4b-8449-689172ef80be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:05.841735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.841658 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-credential-socket" pod "bc8f16d1-4d9c-4c4b-8449-689172ef80be" (UID: "bc8f16d1-4d9c-4c4b-8449-689172ef80be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:05.842016 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.841992 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istiod-ca-cert" pod "bc8f16d1-4d9c-4c4b-8449-689172ef80be" (UID: "bc8f16d1-4d9c-4c4b-8449-689172ef80be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:05.842391 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.842368 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:05.842474 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.842398 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:05.842474 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.842415 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:05.842474 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.842429 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:05.842474 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.842443 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:05.843694 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.843670 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8f16d1-4d9c-4c4b-8449-689172ef80be-kube-api-access-spc47" pod "bc8f16d1-4d9c-4c4b-8449-689172ef80be" (UID: "bc8f16d1-4d9c-4c4b-8449-689172ef80be"). InnerVolumeSpecName "kube-api-access-spc47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:05.843942 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.843925 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-envoy" pod "bc8f16d1-4d9c-4c4b-8449-689172ef80be" (UID: "bc8f16d1-4d9c-4c4b-8449-689172ef80be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:05.844117 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.844092 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-podinfo" pod "bc8f16d1-4d9c-4c4b-8449-689172ef80be" (UID: "bc8f16d1-4d9c-4c4b-8449-689172ef80be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:00:05.844238 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.844122 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-token" pod "bc8f16d1-4d9c-4c4b-8449-689172ef80be" (UID: "bc8f16d1-4d9c-4c4b-8449-689172ef80be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:05.943242 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.943213 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:05.943242 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.943238 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:05.943242 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.943247 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc8f16d1-4d9c-4c4b-8449-689172ef80be-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:05.943535 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:05.943257 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spc47\" (UniqueName: \"kubernetes.io/projected/bc8f16d1-4d9c-4c4b-8449-689172ef80be-kube-api-access-spc47\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:06.471863 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:06.471827 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:06.804651 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:06.804573 5463 scope.go:122] "RemoveContainer" containerID="3f09a62605a10ded604aafea78f64a3d635ee372ce21c9198e89c11c85dfafdc" Nov 03 03:00:06.805058 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:06.804934 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w_odh-ai-gateway-infra(ae465e23-7552-4847-8526-028628c3cb05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" podUID="ae465e23-7552-4847-8526-028628c3cb05" Nov 03 03:00:06.806012 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:06.805984 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh" Nov 03 03:00:06.808068 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:06.808049 5463 scope.go:122] "RemoveContainer" containerID="2081a884d3e1f00ab9a9113842a6489a5b5f56b3eaa28205af58a5d590451afa" Nov 03 03:00:06.808336 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:06.808313 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894580-h8t6n_odh-ai-gateway-infra(510e94f4-1d02-4c4b-8928-6253787c69e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" podUID="510e94f4-1d02-4c4b-8928-6253787c69e4" Nov 03 03:00:07.098993 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:07.098890 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh"] Nov 03 03:00:07.102383 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:07.102363 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ecdbc7-openshift-default-68878fc89f-p49dh"] Nov 03 03:00:07.405147 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:07.405113 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:08.445933 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:08.445900 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc8f16d1-4d9c-4c4b-8449-689172ef80be" path="/var/lib/kubelet/pods/bc8f16d1-4d9c-4c4b-8449-689172ef80be/volumes" Nov 03 03:00:08.895022 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:08.894991 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:09.067502 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:09.067464 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:11.618511 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:11.618471 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:13.007867 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:13.007829 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:00:13.396805 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:13.396771 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:14.445459 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:14.445426 5463 scope.go:122] "RemoveContainer" containerID="2ff4c616b7498af37cbda60a3c1d69dfd09e15f285ad93d3441516c624e106fb" Nov 03 03:00:14.687985 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:14.687957 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2"] Nov 03 03:00:14.688165 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:14.688141 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" podUID="7aa45c20-0b65-4808-80e2-ba8c767966f9" containerName="istio-proxy" containerID="cri-o://d07472982e7835970e7ba007cc30c54df7bf689f5def87d47179f62bd6dbb0d9" gracePeriod=30 Nov 03 03:00:14.873337 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:14.873309 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" event={"ID":"d7ddd193-7cb2-401b-93b9-407faf6ed1fc","Type":"ContainerStarted","Data":"467d5226626250b246b85abb955f764549f6160af5c91c23dfd96c8558db8dbd"} Nov 03 03:00:15.435637 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:15.435605 5463 scope.go:122] "RemoveContainer" containerID="3e1f936b53c2fab544ab871ae962a76e6b52ff0b8d48d401cfbc8d545f93f325" Nov 03 03:00:15.435852 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:15.435783 5463 scope.go:122] "RemoveContainer" containerID="8e04def2f1cc5292821db30d8df8ff164008c2f55fba671e18ed697c56f765e3" Nov 03 03:00:15.882445 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:15.882421 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" event={"ID":"f1300f44-ac64-49c0-9387-8279a643f646","Type":"ContainerStarted","Data":"0a5c26780bb72e486631d72525851937b291b9b5517bf624a936f97e30929ef9"} Nov 03 03:00:15.884410 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:15.884385 5463 generic.go:356] "Generic (PLEG): container finished" podID="d7ddd193-7cb2-401b-93b9-407faf6ed1fc" containerID="467d5226626250b246b85abb955f764549f6160af5c91c23dfd96c8558db8dbd" exitCode=60 Nov 03 03:00:15.884519 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:15.884448 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" event={"ID":"d7ddd193-7cb2-401b-93b9-407faf6ed1fc","Type":"ContainerDied","Data":"467d5226626250b246b85abb955f764549f6160af5c91c23dfd96c8558db8dbd"} Nov 03 03:00:15.884519 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:15.884510 5463 scope.go:122] "RemoveContainer" containerID="2ff4c616b7498af37cbda60a3c1d69dfd09e15f285ad93d3441516c624e106fb" Nov 03 03:00:15.884971 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:15.884953 5463 scope.go:122] "RemoveContainer" containerID="467d5226626250b246b85abb955f764549f6160af5c91c23dfd96c8558db8dbd" Nov 03 03:00:15.885318 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:15.885299 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2_odh-ai-gateway-infra(d7ddd193-7cb2-401b-93b9-407faf6ed1fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" podUID="d7ddd193-7cb2-401b-93b9-407faf6ed1fc" Nov 03 03:00:15.998374 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:15.998340 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2"] Nov 03 03:00:16.431741 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:16.431706 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:16.445204 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:16.445173 5463 scope.go:122] "RemoveContainer" containerID="2577b03ef2e17ad23463cbe20ea6fe5307c7d1e88d254c3ae6011241644e6bc6" Nov 03 03:00:16.894062 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:16.894030 5463 generic.go:356] "Generic (PLEG): container finished" podID="f1300f44-ac64-49c0-9387-8279a643f646" containerID="0a5c26780bb72e486631d72525851937b291b9b5517bf624a936f97e30929ef9" exitCode=60 Nov 03 03:00:16.894567 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:16.894103 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" event={"ID":"f1300f44-ac64-49c0-9387-8279a643f646","Type":"ContainerDied","Data":"0a5c26780bb72e486631d72525851937b291b9b5517bf624a936f97e30929ef9"} Nov 03 03:00:16.894567 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:16.894172 5463 scope.go:122] "RemoveContainer" containerID="8e04def2f1cc5292821db30d8df8ff164008c2f55fba671e18ed697c56f765e3" Nov 03 03:00:16.894567 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:16.894557 5463 scope.go:122] "RemoveContainer" containerID="0a5c26780bb72e486631d72525851937b291b9b5517bf624a936f97e30929ef9" Nov 03 03:00:16.894921 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:16.894896 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv_odh-ai-gateway-infra(f1300f44-ac64-49c0-9387-8279a643f646)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" podUID="f1300f44-ac64-49c0-9387-8279a643f646" Nov 03 03:00:16.896565 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:16.896544 5463 generic.go:356] "Generic (PLEG): container finished" podID="a6e57752-f49f-4fda-a9f7-8f902dfd6fb2" containerID="23f6c2a89a00d29a639f9d512ec907c0450f5e2aa79944ba1379e48f302b2d5b" exitCode=60 Nov 03 03:00:16.896670 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:16.896613 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh" event={"ID":"a6e57752-f49f-4fda-a9f7-8f902dfd6fb2","Type":"ContainerDied","Data":"23f6c2a89a00d29a639f9d512ec907c0450f5e2aa79944ba1379e48f302b2d5b"} Nov 03 03:00:16.896959 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:16.896942 5463 scope.go:122] "RemoveContainer" containerID="23f6c2a89a00d29a639f9d512ec907c0450f5e2aa79944ba1379e48f302b2d5b" Nov 03 03:00:16.897269 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:16.897241 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh_odh-ai-gateway-infra(a6e57752-f49f-4fda-a9f7-8f902dfd6fb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh" podUID="a6e57752-f49f-4fda-a9f7-8f902dfd6fb2" Nov 03 03:00:16.900735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:16.900711 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" event={"ID":"0903eaf9-cfca-4e03-8200-561cff5ab89f","Type":"ContainerStarted","Data":"feaa6b103e2b4ba3a3bd84f1fc97add437bef2c2e81c059c1984d9807c084838"} Nov 03 03:00:16.908962 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:16.908944 5463 scope.go:122] "RemoveContainer" containerID="3e1f936b53c2fab544ab871ae962a76e6b52ff0b8d48d401cfbc8d545f93f325" Nov 03 03:00:17.012699 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:17.012665 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv"] Nov 03 03:00:17.072182 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:17.072154 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" Nov 03 03:00:17.217032 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:17.216954 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7ddd193-7cb2-401b-93b9-407faf6ed1fc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ddd193-7cb2-401b-93b9-407faf6ed1fc-maas-api-service-ca\") pod \"d7ddd193-7cb2-401b-93b9-407faf6ed1fc\" (UID: \"d7ddd193-7cb2-401b-93b9-407faf6ed1fc\") " Nov 03 03:00:17.217175 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:17.217039 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7ddd193-7cb2-401b-93b9-407faf6ed1fc-kube-api-access-bpww8\" (UniqueName: \"kubernetes.io/projected/d7ddd193-7cb2-401b-93b9-407faf6ed1fc-kube-api-access-bpww8\") pod \"d7ddd193-7cb2-401b-93b9-407faf6ed1fc\" (UID: \"d7ddd193-7cb2-401b-93b9-407faf6ed1fc\") " Nov 03 03:00:17.217352 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:17.217323 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ddd193-7cb2-401b-93b9-407faf6ed1fc-maas-api-service-ca" pod "d7ddd193-7cb2-401b-93b9-407faf6ed1fc" (UID: "d7ddd193-7cb2-401b-93b9-407faf6ed1fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:17.217649 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:17.217631 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ddd193-7cb2-401b-93b9-407faf6ed1fc-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:17.219123 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:17.219098 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ddd193-7cb2-401b-93b9-407faf6ed1fc-kube-api-access-bpww8" pod "d7ddd193-7cb2-401b-93b9-407faf6ed1fc" (UID: "d7ddd193-7cb2-401b-93b9-407faf6ed1fc"). InnerVolumeSpecName "kube-api-access-bpww8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:17.319566 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:17.319535 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpww8\" (UniqueName: \"kubernetes.io/projected/d7ddd193-7cb2-401b-93b9-407faf6ed1fc-kube-api-access-bpww8\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:17.435729 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:17.435701 5463 scope.go:122] "RemoveContainer" containerID="2081a884d3e1f00ab9a9113842a6489a5b5f56b3eaa28205af58a5d590451afa" Nov 03 03:00:17.910316 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:17.910288 5463 generic.go:356] "Generic (PLEG): container finished" podID="0903eaf9-cfca-4e03-8200-561cff5ab89f" containerID="feaa6b103e2b4ba3a3bd84f1fc97add437bef2c2e81c059c1984d9807c084838" exitCode=60 Nov 03 03:00:17.910749 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:17.910363 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" event={"ID":"0903eaf9-cfca-4e03-8200-561cff5ab89f","Type":"ContainerDied","Data":"feaa6b103e2b4ba3a3bd84f1fc97add437bef2c2e81c059c1984d9807c084838"} Nov 03 03:00:17.910749 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:17.910429 5463 scope.go:122] "RemoveContainer" containerID="2577b03ef2e17ad23463cbe20ea6fe5307c7d1e88d254c3ae6011241644e6bc6" Nov 03 03:00:17.910889 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:17.910782 5463 scope.go:122] "RemoveContainer" containerID="feaa6b103e2b4ba3a3bd84f1fc97add437bef2c2e81c059c1984d9807c084838" Nov 03 03:00:17.911132 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:17.911101 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6_odh-ai-gateway-infra(0903eaf9-cfca-4e03-8200-561cff5ab89f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" podUID="0903eaf9-cfca-4e03-8200-561cff5ab89f" Nov 03 03:00:17.912582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:17.912559 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" event={"ID":"510e94f4-1d02-4c4b-8928-6253787c69e4","Type":"ContainerStarted","Data":"edb195440dcd9f382286fe6f6c22be0f34b3a384f77b813a65b74369f5843636"} Nov 03 03:00:17.917550 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:17.917531 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" event={"ID":"d7ddd193-7cb2-401b-93b9-407faf6ed1fc","Type":"ContainerDied","Data":"859c55bf7d0858d88881fa674321f40e797d464ac7c117f707280b04b9aa4299"} Nov 03 03:00:17.917639 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:17.917578 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2" Nov 03 03:00:17.931353 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:17.931331 5463 scope.go:122] "RemoveContainer" containerID="467d5226626250b246b85abb955f764549f6160af5c91c23dfd96c8558db8dbd" Nov 03 03:00:18.086616 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.086589 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" Nov 03 03:00:18.149291 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.149260 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6"] Nov 03 03:00:18.243109 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.243068 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1300f44-ac64-49c0-9387-8279a643f646-kube-api-access-6w8rd\" (UniqueName: \"kubernetes.io/projected/f1300f44-ac64-49c0-9387-8279a643f646-kube-api-access-6w8rd\") pod \"f1300f44-ac64-49c0-9387-8279a643f646\" (UID: \"f1300f44-ac64-49c0-9387-8279a643f646\") " Nov 03 03:00:18.243325 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.243271 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1300f44-ac64-49c0-9387-8279a643f646-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1300f44-ac64-49c0-9387-8279a643f646-maas-api-service-ca\") pod \"f1300f44-ac64-49c0-9387-8279a643f646\" (UID: \"f1300f44-ac64-49c0-9387-8279a643f646\") " Nov 03 03:00:18.243617 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.243590 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1300f44-ac64-49c0-9387-8279a643f646-maas-api-service-ca" pod "f1300f44-ac64-49c0-9387-8279a643f646" (UID: "f1300f44-ac64-49c0-9387-8279a643f646"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:18.243894 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.243879 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1300f44-ac64-49c0-9387-8279a643f646-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:18.245245 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.245219 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1300f44-ac64-49c0-9387-8279a643f646-kube-api-access-6w8rd" pod "f1300f44-ac64-49c0-9387-8279a643f646" (UID: "f1300f44-ac64-49c0-9387-8279a643f646"). InnerVolumeSpecName "kube-api-access-6w8rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:18.345974 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.345911 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w8rd\" (UniqueName: \"kubernetes.io/projected/f1300f44-ac64-49c0-9387-8279a643f646-kube-api-access-6w8rd\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:18.367461 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.367433 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2"] Nov 03 03:00:18.370873 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.370850 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9a5e31-29894580-skdj2"] Nov 03 03:00:18.381244 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.381220 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh"] Nov 03 03:00:18.491644 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.491608 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7ddd193-7cb2-401b-93b9-407faf6ed1fc" path="/var/lib/kubelet/pods/d7ddd193-7cb2-401b-93b9-407faf6ed1fc/volumes" Nov 03 03:00:18.493056 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.493032 5463 scope.go:122] "RemoveContainer" containerID="3f09a62605a10ded604aafea78f64a3d635ee372ce21c9198e89c11c85dfafdc" Nov 03 03:00:18.567865 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.567841 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh" Nov 03 03:00:18.753512 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.753472 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e57752-f49f-4fda-a9f7-8f902dfd6fb2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e57752-f49f-4fda-a9f7-8f902dfd6fb2-maas-api-service-ca\") pod \"a6e57752-f49f-4fda-a9f7-8f902dfd6fb2\" (UID: \"a6e57752-f49f-4fda-a9f7-8f902dfd6fb2\") " Nov 03 03:00:18.753734 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.753713 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e57752-f49f-4fda-a9f7-8f902dfd6fb2-kube-api-access-wxlf2\" (UniqueName: \"kubernetes.io/projected/a6e57752-f49f-4fda-a9f7-8f902dfd6fb2-kube-api-access-wxlf2\") pod \"a6e57752-f49f-4fda-a9f7-8f902dfd6fb2\" (UID: \"a6e57752-f49f-4fda-a9f7-8f902dfd6fb2\") " Nov 03 03:00:18.756851 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.756812 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e57752-f49f-4fda-a9f7-8f902dfd6fb2-kube-api-access-wxlf2" pod "a6e57752-f49f-4fda-a9f7-8f902dfd6fb2" (UID: "a6e57752-f49f-4fda-a9f7-8f902dfd6fb2"). InnerVolumeSpecName "kube-api-access-wxlf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:18.757169 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.757148 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e57752-f49f-4fda-a9f7-8f902dfd6fb2-maas-api-service-ca" pod "a6e57752-f49f-4fda-a9f7-8f902dfd6fb2" (UID: "a6e57752-f49f-4fda-a9f7-8f902dfd6fb2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:18.856547 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.856392 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxlf2\" (UniqueName: \"kubernetes.io/projected/a6e57752-f49f-4fda-a9f7-8f902dfd6fb2-kube-api-access-wxlf2\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:18.856547 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.856429 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e57752-f49f-4fda-a9f7-8f902dfd6fb2-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:18.927520 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.927490 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh" Nov 03 03:00:18.927966 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.927488 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh" event={"ID":"a6e57752-f49f-4fda-a9f7-8f902dfd6fb2","Type":"ContainerDied","Data":"ac258face9382ce558aa0fcfb9b787ab6afcf948b175d34afdf4e54ec168761a"} Nov 03 03:00:18.927966 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.927654 5463 scope.go:122] "RemoveContainer" containerID="23f6c2a89a00d29a639f9d512ec907c0450f5e2aa79944ba1379e48f302b2d5b" Nov 03 03:00:18.932861 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.932839 5463 generic.go:356] "Generic (PLEG): container finished" podID="510e94f4-1d02-4c4b-8928-6253787c69e4" containerID="edb195440dcd9f382286fe6f6c22be0f34b3a384f77b813a65b74369f5843636" exitCode=60 Nov 03 03:00:18.932987 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.932869 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" event={"ID":"510e94f4-1d02-4c4b-8928-6253787c69e4","Type":"ContainerDied","Data":"edb195440dcd9f382286fe6f6c22be0f34b3a384f77b813a65b74369f5843636"} Nov 03 03:00:18.933358 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.933338 5463 scope.go:122] "RemoveContainer" containerID="edb195440dcd9f382286fe6f6c22be0f34b3a384f77b813a65b74369f5843636" Nov 03 03:00:18.933803 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:18.933685 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894580-h8t6n_odh-ai-gateway-infra(510e94f4-1d02-4c4b-8928-6253787c69e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" podUID="510e94f4-1d02-4c4b-8928-6253787c69e4" Nov 03 03:00:18.935037 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.935009 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" event={"ID":"f1300f44-ac64-49c0-9387-8279a643f646","Type":"ContainerDied","Data":"ab53fb2319e221aacc6ab4bb3a62931a250e90b8f453e2f6482fca891e552ceb"} Nov 03 03:00:18.935149 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.935107 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv" Nov 03 03:00:18.941751 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.941722 5463 scope.go:122] "RemoveContainer" containerID="2081a884d3e1f00ab9a9113842a6489a5b5f56b3eaa28205af58a5d590451afa" Nov 03 03:00:18.960729 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:18.960256 5463 scope.go:122] "RemoveContainer" containerID="0a5c26780bb72e486631d72525851937b291b9b5517bf624a936f97e30929ef9" Nov 03 03:00:19.116877 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.116850 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" Nov 03 03:00:19.154878 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.154848 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n"] Nov 03 03:00:19.261907 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.261875 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0903eaf9-cfca-4e03-8200-561cff5ab89f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0903eaf9-cfca-4e03-8200-561cff5ab89f-maas-api-service-ca\") pod \"0903eaf9-cfca-4e03-8200-561cff5ab89f\" (UID: \"0903eaf9-cfca-4e03-8200-561cff5ab89f\") " Nov 03 03:00:19.262110 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.262064 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0903eaf9-cfca-4e03-8200-561cff5ab89f-kube-api-access-sscbx\" (UniqueName: \"kubernetes.io/projected/0903eaf9-cfca-4e03-8200-561cff5ab89f-kube-api-access-sscbx\") pod \"0903eaf9-cfca-4e03-8200-561cff5ab89f\" (UID: \"0903eaf9-cfca-4e03-8200-561cff5ab89f\") " Nov 03 03:00:19.262313 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.262291 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0903eaf9-cfca-4e03-8200-561cff5ab89f-maas-api-service-ca" pod "0903eaf9-cfca-4e03-8200-561cff5ab89f" (UID: "0903eaf9-cfca-4e03-8200-561cff5ab89f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:19.262647 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.262628 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0903eaf9-cfca-4e03-8200-561cff5ab89f-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:19.264231 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.264205 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0903eaf9-cfca-4e03-8200-561cff5ab89f-kube-api-access-sscbx" pod "0903eaf9-cfca-4e03-8200-561cff5ab89f" (UID: "0903eaf9-cfca-4e03-8200-561cff5ab89f"). InnerVolumeSpecName "kube-api-access-sscbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:19.325311 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.325278 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv"] Nov 03 03:00:19.331997 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.331937 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7273af-29894580-49fdv"] Nov 03 03:00:19.363551 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.363525 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sscbx\" (UniqueName: \"kubernetes.io/projected/0903eaf9-cfca-4e03-8200-561cff5ab89f-kube-api-access-sscbx\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:19.441622 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.441597 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh"] Nov 03 03:00:19.444709 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.444685 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ddcbd0-29894580-sktrh"] Nov 03 03:00:19.946859 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.946786 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" Nov 03 03:00:19.946859 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.946814 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6" event={"ID":"0903eaf9-cfca-4e03-8200-561cff5ab89f","Type":"ContainerDied","Data":"0c53dafded6a43acb9174bd85744c868d0a7f2f632cfa3bfc5459c128a710a28"} Nov 03 03:00:19.947379 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.946880 5463 scope.go:122] "RemoveContainer" containerID="feaa6b103e2b4ba3a3bd84f1fc97add437bef2c2e81c059c1984d9807c084838" Nov 03 03:00:19.960567 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.960530 5463 generic.go:356] "Generic (PLEG): container finished" podID="7aa45c20-0b65-4808-80e2-ba8c767966f9" containerID="d07472982e7835970e7ba007cc30c54df7bf689f5def87d47179f62bd6dbb0d9" exitCode=0 Nov 03 03:00:19.960692 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.960650 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" event={"ID":"7aa45c20-0b65-4808-80e2-ba8c767966f9","Type":"ContainerDied","Data":"d07472982e7835970e7ba007cc30c54df7bf689f5def87d47179f62bd6dbb0d9"} Nov 03 03:00:19.963322 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.963300 5463 generic.go:356] "Generic (PLEG): container finished" podID="ae465e23-7552-4847-8526-028628c3cb05" containerID="6476a7272a3eb273258c2b478b5990c1223261d56dd2a6f1a75e187579842435" exitCode=60 Nov 03 03:00:19.963465 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.963383 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" event={"ID":"ae465e23-7552-4847-8526-028628c3cb05","Type":"ContainerDied","Data":"6476a7272a3eb273258c2b478b5990c1223261d56dd2a6f1a75e187579842435"} Nov 03 03:00:19.963465 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.963445 5463 scope.go:122] "RemoveContainer" containerID="3f09a62605a10ded604aafea78f64a3d635ee372ce21c9198e89c11c85dfafdc" Nov 03 03:00:19.963894 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:19.963872 5463 scope.go:122] "RemoveContainer" containerID="6476a7272a3eb273258c2b478b5990c1223261d56dd2a6f1a75e187579842435" Nov 03 03:00:19.964321 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:19.964296 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w_odh-ai-gateway-infra(ae465e23-7552-4847-8526-028628c3cb05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" podUID="ae465e23-7552-4847-8526-028628c3cb05" Nov 03 03:00:20.001180 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.000856 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 03:00:20.085238 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.084330 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-credential-socket\") pod \"7aa45c20-0b65-4808-80e2-ba8c767966f9\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " Nov 03 03:00:20.085238 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.084401 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-envoy\") pod \"7aa45c20-0b65-4808-80e2-ba8c767966f9\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " Nov 03 03:00:20.085238 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.084472 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-data\") pod \"7aa45c20-0b65-4808-80e2-ba8c767966f9\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " Nov 03 03:00:20.085238 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.084550 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-workload-certs\") pod \"7aa45c20-0b65-4808-80e2-ba8c767966f9\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " Nov 03 03:00:20.085238 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.084686 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-token\" (UniqueName: \"kubernetes.io/projected/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-token\") pod \"7aa45c20-0b65-4808-80e2-ba8c767966f9\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " Nov 03 03:00:20.085238 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.084721 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-workload-socket\") pod \"7aa45c20-0b65-4808-80e2-ba8c767966f9\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " Nov 03 03:00:20.085238 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.084779 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa45c20-0b65-4808-80e2-ba8c767966f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa45c20-0b65-4808-80e2-ba8c767966f9-istiod-ca-cert\") pod \"7aa45c20-0b65-4808-80e2-ba8c767966f9\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " Nov 03 03:00:20.085238 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.084882 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa45c20-0b65-4808-80e2-ba8c767966f9-kube-api-access-fchl4\" (UniqueName: \"kubernetes.io/projected/7aa45c20-0b65-4808-80e2-ba8c767966f9-kube-api-access-fchl4\") pod \"7aa45c20-0b65-4808-80e2-ba8c767966f9\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " Nov 03 03:00:20.085238 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.084948 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-podinfo\") pod \"7aa45c20-0b65-4808-80e2-ba8c767966f9\" (UID: \"7aa45c20-0b65-4808-80e2-ba8c767966f9\") " Nov 03 03:00:20.086512 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.086343 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-credential-socket" pod "7aa45c20-0b65-4808-80e2-ba8c767966f9" (UID: "7aa45c20-0b65-4808-80e2-ba8c767966f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:20.086821 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.086647 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-workload-certs" pod "7aa45c20-0b65-4808-80e2-ba8c767966f9" (UID: "7aa45c20-0b65-4808-80e2-ba8c767966f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:20.086821 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.086661 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-workload-socket" pod "7aa45c20-0b65-4808-80e2-ba8c767966f9" (UID: "7aa45c20-0b65-4808-80e2-ba8c767966f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:20.086946 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.086862 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-data" pod "7aa45c20-0b65-4808-80e2-ba8c767966f9" (UID: "7aa45c20-0b65-4808-80e2-ba8c767966f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:20.087521 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.087180 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa45c20-0b65-4808-80e2-ba8c767966f9-istiod-ca-cert" pod "7aa45c20-0b65-4808-80e2-ba8c767966f9" (UID: "7aa45c20-0b65-4808-80e2-ba8c767966f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:20.090165 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.090129 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-token" pod "7aa45c20-0b65-4808-80e2-ba8c767966f9" (UID: "7aa45c20-0b65-4808-80e2-ba8c767966f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:20.090554 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.090533 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-podinfo" pod "7aa45c20-0b65-4808-80e2-ba8c767966f9" (UID: "7aa45c20-0b65-4808-80e2-ba8c767966f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:00:20.091629 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.091598 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa45c20-0b65-4808-80e2-ba8c767966f9-kube-api-access-fchl4" pod "7aa45c20-0b65-4808-80e2-ba8c767966f9" (UID: "7aa45c20-0b65-4808-80e2-ba8c767966f9"). InnerVolumeSpecName "kube-api-access-fchl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:20.092386 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.092364 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-envoy" pod "7aa45c20-0b65-4808-80e2-ba8c767966f9" (UID: "7aa45c20-0b65-4808-80e2-ba8c767966f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:20.116074 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.116048 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" Nov 03 03:00:20.186201 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.186156 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/510e94f4-1d02-4c4b-8928-6253787c69e4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/510e94f4-1d02-4c4b-8928-6253787c69e4-maas-api-service-ca\") pod \"510e94f4-1d02-4c4b-8928-6253787c69e4\" (UID: \"510e94f4-1d02-4c4b-8928-6253787c69e4\") " Nov 03 03:00:20.186354 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.186254 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510e94f4-1d02-4c4b-8928-6253787c69e4-kube-api-access-m9n4t\" (UniqueName: \"kubernetes.io/projected/510e94f4-1d02-4c4b-8928-6253787c69e4-kube-api-access-m9n4t\") pod \"510e94f4-1d02-4c4b-8928-6253787c69e4\" (UID: \"510e94f4-1d02-4c4b-8928-6253787c69e4\") " Nov 03 03:00:20.186583 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.186550 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/510e94f4-1d02-4c4b-8928-6253787c69e4-maas-api-service-ca" pod "510e94f4-1d02-4c4b-8928-6253787c69e4" (UID: "510e94f4-1d02-4c4b-8928-6253787c69e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:20.186860 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.186842 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa45c20-0b65-4808-80e2-ba8c767966f9-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.186927 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.186866 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fchl4\" (UniqueName: \"kubernetes.io/projected/7aa45c20-0b65-4808-80e2-ba8c767966f9-kube-api-access-fchl4\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.186927 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.186882 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.186927 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.186894 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.186927 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.186903 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.186927 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.186911 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.186927 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.186920 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/510e94f4-1d02-4c4b-8928-6253787c69e4-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.186927 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.186928 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.187315 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.186936 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa45c20-0b65-4808-80e2-ba8c767966f9-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.187315 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.186945 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa45c20-0b65-4808-80e2-ba8c767966f9-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.188395 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.188365 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510e94f4-1d02-4c4b-8928-6253787c69e4-kube-api-access-m9n4t" pod "510e94f4-1d02-4c4b-8928-6253787c69e4" (UID: "510e94f4-1d02-4c4b-8928-6253787c69e4"). InnerVolumeSpecName "kube-api-access-m9n4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:20.238041 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.237969 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6"] Nov 03 03:00:20.241320 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.241299 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7d758-29894580-tv8n6"] Nov 03 03:00:20.288931 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.288899 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9n4t\" (UniqueName: \"kubernetes.io/projected/510e94f4-1d02-4c4b-8928-6253787c69e4-kube-api-access-m9n4t\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:20.445370 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.445334 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0903eaf9-cfca-4e03-8200-561cff5ab89f" path="/var/lib/kubelet/pods/0903eaf9-cfca-4e03-8200-561cff5ab89f/volumes" Nov 03 03:00:20.445782 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.445767 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e57752-f49f-4fda-a9f7-8f902dfd6fb2" path="/var/lib/kubelet/pods/a6e57752-f49f-4fda-a9f7-8f902dfd6fb2/volumes" Nov 03 03:00:20.446250 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.446227 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1300f44-ac64-49c0-9387-8279a643f646" path="/var/lib/kubelet/pods/f1300f44-ac64-49c0-9387-8279a643f646/volumes" Nov 03 03:00:20.977325 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.977285 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" Nov 03 03:00:20.977831 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.977283 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n" event={"ID":"510e94f4-1d02-4c4b-8928-6253787c69e4","Type":"ContainerDied","Data":"ce35ba4c5a97d0a5a1ce8740e4cfbd3e3a57f1fa15e114bcfbe3fa2221e7a5e3"} Nov 03 03:00:20.977831 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.977504 5463 scope.go:122] "RemoveContainer" containerID="edb195440dcd9f382286fe6f6c22be0f34b3a384f77b813a65b74369f5843636" Nov 03 03:00:20.979278 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.979248 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" event={"ID":"7aa45c20-0b65-4808-80e2-ba8c767966f9","Type":"ContainerDied","Data":"d89beba393621fcaa6c3125e48ed8b6a8397ab332eb745de134325169fd27472"} Nov 03 03:00:20.979392 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.979340 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2" Nov 03 03:00:20.991329 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:20.991284 5463 scope.go:122] "RemoveContainer" containerID="d07472982e7835970e7ba007cc30c54df7bf689f5def87d47179f62bd6dbb0d9" Nov 03 03:00:21.150244 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:21.150211 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2"] Nov 03 03:00:21.199716 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:21.199687 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-pbnr2"] Nov 03 03:00:21.210361 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:21.210326 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:21.308815 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:21.308732 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n"] Nov 03 03:00:21.312637 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:21.312612 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894580-h8t6n"] Nov 03 03:00:21.481291 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:21.481250 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w"] Nov 03 03:00:21.652687 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:21.652659 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" Nov 03 03:00:21.713591 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:21.713559 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae465e23-7552-4847-8526-028628c3cb05-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae465e23-7552-4847-8526-028628c3cb05-maas-api-service-ca\") pod \"ae465e23-7552-4847-8526-028628c3cb05\" (UID: \"ae465e23-7552-4847-8526-028628c3cb05\") " Nov 03 03:00:21.713744 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:21.713655 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae465e23-7552-4847-8526-028628c3cb05-kube-api-access-vc4lk\" (UniqueName: \"kubernetes.io/projected/ae465e23-7552-4847-8526-028628c3cb05-kube-api-access-vc4lk\") pod \"ae465e23-7552-4847-8526-028628c3cb05\" (UID: \"ae465e23-7552-4847-8526-028628c3cb05\") " Nov 03 03:00:21.714035 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:21.714001 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae465e23-7552-4847-8526-028628c3cb05-maas-api-service-ca" pod "ae465e23-7552-4847-8526-028628c3cb05" (UID: "ae465e23-7552-4847-8526-028628c3cb05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:21.714224 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:21.714171 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae465e23-7552-4847-8526-028628c3cb05-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:21.715799 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:21.715781 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae465e23-7552-4847-8526-028628c3cb05-kube-api-access-vc4lk" pod "ae465e23-7552-4847-8526-028628c3cb05" (UID: "ae465e23-7552-4847-8526-028628c3cb05"). InnerVolumeSpecName "kube-api-access-vc4lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:21.816159 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:21.816115 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc4lk\" (UniqueName: \"kubernetes.io/projected/ae465e23-7552-4847-8526-028628c3cb05-kube-api-access-vc4lk\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:22.000419 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:22.000388 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" Nov 03 03:00:22.000832 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:22.000391 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w" event={"ID":"ae465e23-7552-4847-8526-028628c3cb05","Type":"ContainerDied","Data":"d5a22c16ceb59a99a263720272e398f0545836bed9040fa94af2ee358981f3ae"} Nov 03 03:00:22.000832 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:22.000518 5463 scope.go:122] "RemoveContainer" containerID="6476a7272a3eb273258c2b478b5990c1223261d56dd2a6f1a75e187579842435" Nov 03 03:00:22.165588 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:22.165559 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w"] Nov 03 03:00:22.168791 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:22.168764 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ffba43-29894580-7h99w"] Nov 03 03:00:22.443418 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:22.443386 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="510e94f4-1d02-4c4b-8928-6253787c69e4" path="/var/lib/kubelet/pods/510e94f4-1d02-4c4b-8928-6253787c69e4/volumes" Nov 03 03:00:22.443890 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:22.443876 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa45c20-0b65-4808-80e2-ba8c767966f9" path="/var/lib/kubelet/pods/7aa45c20-0b65-4808-80e2-ba8c767966f9/volumes" Nov 03 03:00:22.444269 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:22.444255 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae465e23-7552-4847-8526-028628c3cb05" path="/var/lib/kubelet/pods/ae465e23-7552-4847-8526-028628c3cb05/volumes" Nov 03 03:00:25.634240 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:25.634180 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:25.959500 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:25.959464 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:30.841588 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:30.841556 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:35.545375 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:35.545340 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:39.562622 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:39.562582 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks"] Nov 03 03:00:39.563119 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:39.562791 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" podUID="c2475661-edad-4611-b2bc-5930241596fa" containerName="maas-api" containerID="cri-o://8d189f9260a464569a2f04b0c0a5f687214c31ac84bb6c0ceb377f75feb30834" gracePeriod=30 Nov 03 03:00:39.681879 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:39.681838 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:39.867609 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:39.867580 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" Nov 03 03:00:39.975212 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:39.972109 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2475661-edad-4611-b2bc-5930241596fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2475661-edad-4611-b2bc-5930241596fa-maas-api-tls\") pod \"c2475661-edad-4611-b2bc-5930241596fa\" (UID: \"c2475661-edad-4611-b2bc-5930241596fa\") " Nov 03 03:00:39.985211 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:39.975660 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2475661-edad-4611-b2bc-5930241596fa-kube-api-access-hvp72\" (UniqueName: \"kubernetes.io/projected/c2475661-edad-4611-b2bc-5930241596fa-kube-api-access-hvp72\") pod \"c2475661-edad-4611-b2bc-5930241596fa\" (UID: \"c2475661-edad-4611-b2bc-5930241596fa\") " Nov 03 03:00:39.999346 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:39.999321 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2475661-edad-4611-b2bc-5930241596fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2475661-edad-4611-b2bc-5930241596fa-openshift-service-ca\") pod \"c2475661-edad-4611-b2bc-5930241596fa\" (UID: \"c2475661-edad-4611-b2bc-5930241596fa\") " Nov 03 03:00:39.999532 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:39.999518 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2475661-edad-4611-b2bc-5930241596fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2475661-edad-4611-b2bc-5930241596fa-metrics-certs\") pod \"c2475661-edad-4611-b2bc-5930241596fa\" (UID: \"c2475661-edad-4611-b2bc-5930241596fa\") " Nov 03 03:00:40.000245 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:39.988313 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2475661-edad-4611-b2bc-5930241596fa-maas-api-tls" pod "c2475661-edad-4611-b2bc-5930241596fa" (UID: "c2475661-edad-4611-b2bc-5930241596fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:00:40.009210 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:40.001030 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2475661-edad-4611-b2bc-5930241596fa-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:40.009580 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:40.001671 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2475661-edad-4611-b2bc-5930241596fa-kube-api-access-hvp72" pod "c2475661-edad-4611-b2bc-5930241596fa" (UID: "c2475661-edad-4611-b2bc-5930241596fa"). InnerVolumeSpecName "kube-api-access-hvp72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:40.009736 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:40.003727 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2475661-edad-4611-b2bc-5930241596fa-metrics-certs" pod "c2475661-edad-4611-b2bc-5930241596fa" (UID: "c2475661-edad-4611-b2bc-5930241596fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:00:40.009822 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:40.009527 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2475661-edad-4611-b2bc-5930241596fa-openshift-service-ca" pod "c2475661-edad-4611-b2bc-5930241596fa" (UID: "c2475661-edad-4611-b2bc-5930241596fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:40.123227 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:40.122957 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvp72\" (UniqueName: \"kubernetes.io/projected/c2475661-edad-4611-b2bc-5930241596fa-kube-api-access-hvp72\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:40.123572 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:40.123543 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2475661-edad-4611-b2bc-5930241596fa-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:40.123712 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:40.123577 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2475661-edad-4611-b2bc-5930241596fa-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:40.145077 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:40.145039 5463 generic.go:356] "Generic (PLEG): container finished" podID="c2475661-edad-4611-b2bc-5930241596fa" containerID="8d189f9260a464569a2f04b0c0a5f687214c31ac84bb6c0ceb377f75feb30834" exitCode=0 Nov 03 03:00:40.145251 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:40.145085 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" Nov 03 03:00:40.145251 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:40.145117 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" event={"ID":"c2475661-edad-4611-b2bc-5930241596fa","Type":"ContainerDied","Data":"8d189f9260a464569a2f04b0c0a5f687214c31ac84bb6c0ceb377f75feb30834"} Nov 03 03:00:40.145251 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:40.145171 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks" event={"ID":"c2475661-edad-4611-b2bc-5930241596fa","Type":"ContainerDied","Data":"fc04d15d2fea20eafc12140b167cf87d872bbdde4ed84d2a2aa42fad5d5e7825"} Nov 03 03:00:40.145251 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:40.145219 5463 scope.go:122] "RemoveContainer" containerID="8d189f9260a464569a2f04b0c0a5f687214c31ac84bb6c0ceb377f75feb30834" Nov 03 03:00:40.158505 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:40.158486 5463 scope.go:122] "RemoveContainer" containerID="8d189f9260a464569a2f04b0c0a5f687214c31ac84bb6c0ceb377f75feb30834" Nov 03 03:00:40.158798 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:40.158771 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d189f9260a464569a2f04b0c0a5f687214c31ac84bb6c0ceb377f75feb30834\": container with ID starting with 8d189f9260a464569a2f04b0c0a5f687214c31ac84bb6c0ceb377f75feb30834 not found: ID does not exist" containerID="8d189f9260a464569a2f04b0c0a5f687214c31ac84bb6c0ceb377f75feb30834" Nov 03 03:00:40.158890 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:40.158804 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d189f9260a464569a2f04b0c0a5f687214c31ac84bb6c0ceb377f75feb30834"} err="failed to get container status \"8d189f9260a464569a2f04b0c0a5f687214c31ac84bb6c0ceb377f75feb30834\": rpc error: code = NotFound desc = could not find container \"8d189f9260a464569a2f04b0c0a5f687214c31ac84bb6c0ceb377f75feb30834\": container with ID starting with 8d189f9260a464569a2f04b0c0a5f687214c31ac84bb6c0ceb377f75feb30834 not found: ID does not exist" Nov 03 03:00:40.311110 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:40.311076 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks"] Nov 03 03:00:40.316063 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:40.316039 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc14f6-54bbb686b4-7hcks"] Nov 03 03:00:40.444448 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:40.444416 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2475661-edad-4611-b2bc-5930241596fa" path="/var/lib/kubelet/pods/c2475661-edad-4611-b2bc-5930241596fa/volumes" Nov 03 03:00:41.383380 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:41.383343 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:41.571523 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:41.571486 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:42.012051 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:42.012020 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr"] Nov 03 03:00:42.012257 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:42.012234 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" podUID="3b38f888-2173-4d9b-a4cb-047d5086e894" containerName="istio-proxy" containerID="cri-o://063f83d315206498c945de2c3c9d040232bf2c41e159e00558294db6967123d4" gracePeriod=30 Nov 03 03:00:42.074891 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:42.074829 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6"] Nov 03 03:00:42.075565 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:42.075489 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" podUID="29bcbc40-df9d-489e-9a7d-50f2c49c8312" containerName="istio-proxy" containerID="cri-o://a0f81cfa8976b39a2e878a2be98d7af948c67b730420ad1acffcbca92c0e2faa" gracePeriod=30 Nov 03 03:00:42.078795 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:42.078771 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:43.010959 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:00:43.010924 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:00:43.048359 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:43.048328 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:43.209241 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:43.209176 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:44.670673 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:44.670640 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:44.900694 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:44.900658 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:46.750041 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:46.750010 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:47.225468 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.225440 5463 generic.go:356] "Generic (PLEG): container finished" podID="3b38f888-2173-4d9b-a4cb-047d5086e894" containerID="063f83d315206498c945de2c3c9d040232bf2c41e159e00558294db6967123d4" exitCode=0 Nov 03 03:00:47.225633 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.225518 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" event={"ID":"3b38f888-2173-4d9b-a4cb-047d5086e894","Type":"ContainerDied","Data":"063f83d315206498c945de2c3c9d040232bf2c41e159e00558294db6967123d4"} Nov 03 03:00:47.225633 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.225597 5463 scope.go:122] "RemoveContainer" containerID="66e900090ec0db270731d1f9c0a91e33778a6d64c6f796dbb40a6cdb762dce19" Nov 03 03:00:47.227511 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.227486 5463 generic.go:356] "Generic (PLEG): container finished" podID="29bcbc40-df9d-489e-9a7d-50f2c49c8312" containerID="a0f81cfa8976b39a2e878a2be98d7af948c67b730420ad1acffcbca92c0e2faa" exitCode=0 Nov 03 03:00:47.227709 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.227692 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" event={"ID":"29bcbc40-df9d-489e-9a7d-50f2c49c8312","Type":"ContainerDied","Data":"a0f81cfa8976b39a2e878a2be98d7af948c67b730420ad1acffcbca92c0e2faa"} Nov 03 03:00:47.384800 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.384777 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 03:00:47.445427 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.445329 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b38f888-2173-4d9b-a4cb-047d5086e894-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b38f888-2173-4d9b-a4cb-047d5086e894-istiod-ca-cert\") pod \"3b38f888-2173-4d9b-a4cb-047d5086e894\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " Nov 03 03:00:47.445427 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.445378 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-workload-certs\") pod \"3b38f888-2173-4d9b-a4cb-047d5086e894\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " Nov 03 03:00:47.446100 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.445462 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-podinfo\") pod \"3b38f888-2173-4d9b-a4cb-047d5086e894\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " Nov 03 03:00:47.446100 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.445553 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-credential-socket\") pod \"3b38f888-2173-4d9b-a4cb-047d5086e894\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " Nov 03 03:00:47.446100 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.445618 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-data\") pod \"3b38f888-2173-4d9b-a4cb-047d5086e894\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " Nov 03 03:00:47.446100 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.445676 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-token\" (UniqueName: \"kubernetes.io/projected/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-token\") pod \"3b38f888-2173-4d9b-a4cb-047d5086e894\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " Nov 03 03:00:47.446100 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.445723 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b38f888-2173-4d9b-a4cb-047d5086e894-kube-api-access-th7kw\" (UniqueName: \"kubernetes.io/projected/3b38f888-2173-4d9b-a4cb-047d5086e894-kube-api-access-th7kw\") pod \"3b38f888-2173-4d9b-a4cb-047d5086e894\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " Nov 03 03:00:47.446100 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.445728 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b38f888-2173-4d9b-a4cb-047d5086e894-istiod-ca-cert" pod "3b38f888-2173-4d9b-a4cb-047d5086e894" (UID: "3b38f888-2173-4d9b-a4cb-047d5086e894"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:47.446100 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.445768 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-workload-socket\") pod \"3b38f888-2173-4d9b-a4cb-047d5086e894\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " Nov 03 03:00:47.446100 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.445851 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-envoy\") pod \"3b38f888-2173-4d9b-a4cb-047d5086e894\" (UID: \"3b38f888-2173-4d9b-a4cb-047d5086e894\") " Nov 03 03:00:47.446100 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.445999 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-credential-socket" pod "3b38f888-2173-4d9b-a4cb-047d5086e894" (UID: "3b38f888-2173-4d9b-a4cb-047d5086e894"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:47.446677 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.446458 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-workload-certs" pod "3b38f888-2173-4d9b-a4cb-047d5086e894" (UID: "3b38f888-2173-4d9b-a4cb-047d5086e894"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:47.447876 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.446464 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b38f888-2173-4d9b-a4cb-047d5086e894-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.447876 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.447822 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.449456 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.449363 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-workload-socket" pod "3b38f888-2173-4d9b-a4cb-047d5086e894" (UID: "3b38f888-2173-4d9b-a4cb-047d5086e894"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:47.449456 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.449432 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-data" pod "3b38f888-2173-4d9b-a4cb-047d5086e894" (UID: "3b38f888-2173-4d9b-a4cb-047d5086e894"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:47.452770 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.452624 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-envoy" pod "3b38f888-2173-4d9b-a4cb-047d5086e894" (UID: "3b38f888-2173-4d9b-a4cb-047d5086e894"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:47.452770 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.452712 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b38f888-2173-4d9b-a4cb-047d5086e894-kube-api-access-th7kw" pod "3b38f888-2173-4d9b-a4cb-047d5086e894" (UID: "3b38f888-2173-4d9b-a4cb-047d5086e894"). InnerVolumeSpecName "kube-api-access-th7kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:47.453003 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.452980 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-podinfo" pod "3b38f888-2173-4d9b-a4cb-047d5086e894" (UID: "3b38f888-2173-4d9b-a4cb-047d5086e894"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:00:47.453794 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.453771 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-token" pod "3b38f888-2173-4d9b-a4cb-047d5086e894" (UID: "3b38f888-2173-4d9b-a4cb-047d5086e894"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:47.465950 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.465929 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 03:00:47.548674 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.548642 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-workload-socket\") pod \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " Nov 03 03:00:47.548891 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.548871 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-token\" (UniqueName: \"kubernetes.io/projected/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-token\") pod \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " Nov 03 03:00:47.548966 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.548947 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-envoy\") pod \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " Nov 03 03:00:47.548966 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.548949 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-workload-socket" pod "29bcbc40-df9d-489e-9a7d-50f2c49c8312" (UID: "29bcbc40-df9d-489e-9a7d-50f2c49c8312"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:47.549088 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.549003 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29bcbc40-df9d-489e-9a7d-50f2c49c8312-kube-api-access-cls9c\" (UniqueName: \"kubernetes.io/projected/29bcbc40-df9d-489e-9a7d-50f2c49c8312-kube-api-access-cls9c\") pod \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " Nov 03 03:00:47.549088 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.549034 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-credential-socket\") pod \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " Nov 03 03:00:47.549088 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.549063 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-podinfo\") pod \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " Nov 03 03:00:47.549321 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.549090 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istiod-ca-cert\") pod \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " Nov 03 03:00:47.549321 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.549159 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-workload-certs\") pod \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " Nov 03 03:00:47.549321 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.549225 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-data\") pod \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\" (UID: \"29bcbc40-df9d-489e-9a7d-50f2c49c8312\") " Nov 03 03:00:47.552233 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.549863 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.552233 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.549890 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.552233 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.549906 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.552233 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.549920 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.552233 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.549934 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.552233 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.549949 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.552233 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.549973 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b38f888-2173-4d9b-a4cb-047d5086e894-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.552233 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.549987 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th7kw\" (UniqueName: \"kubernetes.io/projected/3b38f888-2173-4d9b-a4cb-047d5086e894-kube-api-access-th7kw\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.552233 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.550266 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-workload-certs" pod "29bcbc40-df9d-489e-9a7d-50f2c49c8312" (UID: "29bcbc40-df9d-489e-9a7d-50f2c49c8312"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:47.552233 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.551653 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-token" pod "29bcbc40-df9d-489e-9a7d-50f2c49c8312" (UID: "29bcbc40-df9d-489e-9a7d-50f2c49c8312"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:47.552233 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.551795 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-envoy" pod "29bcbc40-df9d-489e-9a7d-50f2c49c8312" (UID: "29bcbc40-df9d-489e-9a7d-50f2c49c8312"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:47.552851 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.552522 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-podinfo" pod "29bcbc40-df9d-489e-9a7d-50f2c49c8312" (UID: "29bcbc40-df9d-489e-9a7d-50f2c49c8312"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:00:47.553091 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.553057 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istiod-ca-cert" pod "29bcbc40-df9d-489e-9a7d-50f2c49c8312" (UID: "29bcbc40-df9d-489e-9a7d-50f2c49c8312"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:47.553514 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.553267 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-data" pod "29bcbc40-df9d-489e-9a7d-50f2c49c8312" (UID: "29bcbc40-df9d-489e-9a7d-50f2c49c8312"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:47.553683 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.553353 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-credential-socket" pod "29bcbc40-df9d-489e-9a7d-50f2c49c8312" (UID: "29bcbc40-df9d-489e-9a7d-50f2c49c8312"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:00:47.557911 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.556000 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29bcbc40-df9d-489e-9a7d-50f2c49c8312-kube-api-access-cls9c" pod "29bcbc40-df9d-489e-9a7d-50f2c49c8312" (UID: "29bcbc40-df9d-489e-9a7d-50f2c49c8312"). InnerVolumeSpecName "kube-api-access-cls9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:47.651094 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.651061 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.651094 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.651092 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.651294 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.651103 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cls9c\" (UniqueName: \"kubernetes.io/projected/29bcbc40-df9d-489e-9a7d-50f2c49c8312-kube-api-access-cls9c\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.651294 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.651113 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.651294 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.651122 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.651294 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.651131 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.651294 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.651140 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:47.651294 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:47.651148 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29bcbc40-df9d-489e-9a7d-50f2c49c8312-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:48.236562 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:48.236529 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" Nov 03 03:00:48.236562 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:48.236558 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr" event={"ID":"3b38f888-2173-4d9b-a4cb-047d5086e894","Type":"ContainerDied","Data":"4d8561150ed785d50c0ff0fc5df84bde3fc42a76ecbf80c6e572306b5506d283"} Nov 03 03:00:48.237080 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:48.236621 5463 scope.go:122] "RemoveContainer" containerID="063f83d315206498c945de2c3c9d040232bf2c41e159e00558294db6967123d4" Nov 03 03:00:48.238143 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:48.238124 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" event={"ID":"29bcbc40-df9d-489e-9a7d-50f2c49c8312","Type":"ContainerDied","Data":"5d87659e06b5eaf580b804b2691fbb67338ba3b6f3720bc9e26f0b083cc59f10"} Nov 03 03:00:48.238275 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:48.238263 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6" Nov 03 03:00:48.248305 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:48.248288 5463 scope.go:122] "RemoveContainer" containerID="a0f81cfa8976b39a2e878a2be98d7af948c67b730420ad1acffcbca92c0e2faa" Nov 03 03:00:48.403578 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:48.403542 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr"] Nov 03 03:00:48.408863 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:48.408839 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-j8phr"] Nov 03 03:00:48.445185 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:48.445156 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b38f888-2173-4d9b-a4cb-047d5086e894" path="/var/lib/kubelet/pods/3b38f888-2173-4d9b-a4cb-047d5086e894/volumes" Nov 03 03:00:48.565420 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:48.565382 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6"] Nov 03 03:00:48.568494 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:48.568470 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-wd7x6"] Nov 03 03:00:48.946316 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:48.946267 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:49.124838 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:49.124799 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:50.444312 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:50.444273 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29bcbc40-df9d-489e-9a7d-50f2c49c8312" path="/var/lib/kubelet/pods/29bcbc40-df9d-489e-9a7d-50f2c49c8312/volumes" Nov 03 03:00:51.551582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:51.551550 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:56.331991 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:56.331957 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:57.632778 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:57.632728 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:58.480122 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:58.480083 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:58.764256 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:00:58.764147 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:01.138683 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:01.138642 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:03.903108 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:03.903075 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:05.258202 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:05.258152 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:05.623094 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:05.623011 5463 scope.go:122] "RemoveContainer" containerID="20a96e641da37d8465242c1bb60de97266b7ec6ef9f0d73d2196bdaf659212e0" Nov 03 03:01:05.968539 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:05.968505 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:09.390782 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:09.390694 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:10.453953 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:10.453911 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:12.251769 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:12.251726 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:13.012968 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:01:13.012921 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:01:14.964820 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:14.964784 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:15.144458 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:15.144424 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:15.827796 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:15.827765 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:16.137038 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:16.136960 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:16.581913 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:16.581861 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:17.120167 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:17.120125 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:17.393099 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:17.392970 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:18.128378 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:18.128322 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:18.297998 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:18.297950 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:18.493352 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:18.493320 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:19.618011 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:19.617971 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:19.768624 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:19.768587 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:21.623417 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:21.623383 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd"] Nov 03 03:01:21.623863 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:21.623563 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" podUID="bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e" containerName="maas-api" containerID="cri-o://48219a1e4243f24aca80055c384540cf2cb01a102e390bed002804b0c6ba642b" gracePeriod=30 Nov 03 03:01:21.740055 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:21.739982 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:21.905240 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:21.905205 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:21.922128 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:21.922106 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" Nov 03 03:01:22.025628 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.025593 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-maas-api-tls\") pod \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\" (UID: \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\") " Nov 03 03:01:22.025856 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.025710 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-kube-api-access-k2c6l\" (UniqueName: \"kubernetes.io/projected/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-kube-api-access-k2c6l\") pod \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\" (UID: \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\") " Nov 03 03:01:22.025856 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.025781 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-metrics-certs\") pod \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\" (UID: \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\") " Nov 03 03:01:22.025856 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.025848 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-openshift-service-ca\") pod \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\" (UID: \"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e\") " Nov 03 03:01:22.026837 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.026807 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-openshift-service-ca" pod "bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e" (UID: "bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:01:22.034595 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.034562 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-kube-api-access-k2c6l" pod "bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e" (UID: "bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e"). InnerVolumeSpecName "kube-api-access-k2c6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:01:22.037348 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.037306 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-maas-api-tls" pod "bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e" (UID: "bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:01:22.040782 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.040756 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-metrics-certs" pod "bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e" (UID: "bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:01:22.128454 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.128425 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:22.128454 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.128452 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2c6l\" (UniqueName: \"kubernetes.io/projected/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-kube-api-access-k2c6l\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:22.128663 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.128463 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:22.128663 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.128473 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:22.621226 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.619605 5463 generic.go:356] "Generic (PLEG): container finished" podID="bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e" containerID="48219a1e4243f24aca80055c384540cf2cb01a102e390bed002804b0c6ba642b" exitCode=0 Nov 03 03:01:22.621226 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.619692 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" event={"ID":"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e","Type":"ContainerDied","Data":"48219a1e4243f24aca80055c384540cf2cb01a102e390bed002804b0c6ba642b"} Nov 03 03:01:22.621226 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.619741 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" event={"ID":"bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e","Type":"ContainerDied","Data":"e9b2ca2a79b3f4f59239201204ef08c32e81151c890ea1aed1195fd69732dd7c"} Nov 03 03:01:22.621226 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.619771 5463 scope.go:122] "RemoveContainer" containerID="48219a1e4243f24aca80055c384540cf2cb01a102e390bed002804b0c6ba642b" Nov 03 03:01:22.621226 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.619936 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" Nov 03 03:01:22.638752 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.638726 5463 scope.go:122] "RemoveContainer" containerID="48219a1e4243f24aca80055c384540cf2cb01a102e390bed002804b0c6ba642b" Nov 03 03:01:22.639149 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:01:22.639126 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48219a1e4243f24aca80055c384540cf2cb01a102e390bed002804b0c6ba642b\": container with ID starting with 48219a1e4243f24aca80055c384540cf2cb01a102e390bed002804b0c6ba642b not found: ID does not exist" containerID="48219a1e4243f24aca80055c384540cf2cb01a102e390bed002804b0c6ba642b" Nov 03 03:01:22.639262 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.639161 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48219a1e4243f24aca80055c384540cf2cb01a102e390bed002804b0c6ba642b"} err="failed to get container status \"48219a1e4243f24aca80055c384540cf2cb01a102e390bed002804b0c6ba642b\": rpc error: code = NotFound desc = could not find container \"48219a1e4243f24aca80055c384540cf2cb01a102e390bed002804b0c6ba642b\": container with ID starting with 48219a1e4243f24aca80055c384540cf2cb01a102e390bed002804b0c6ba642b not found: ID does not exist" Nov 03 03:01:22.787033 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.786949 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd"] Nov 03 03:01:22.790477 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:22.790454 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd"] Nov 03 03:01:23.548906 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:23.548873 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:24.443897 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:24.443854 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e" path="/var/lib/kubelet/pods/bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e/volumes" Nov 03 03:01:24.567594 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:24.567558 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:24.735985 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:24.735905 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:24.821780 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:24.821741 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a5e31-6885cfbc9d-l5kxd" podUID="bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.23:8443/health\": context deadline exceeded" Nov 03 03:01:25.977059 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:25.977017 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:26.635734 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:26.635703 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:28.333409 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:28.333357 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:31.010604 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:31.010573 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:31.185367 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:31.185336 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:32.612202 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:32.612151 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:33.105836 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:33.105799 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd"] Nov 03 03:01:33.106077 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:33.106048 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" podUID="0d32bca4-00c3-4968-92b0-86c8ed0738c9" containerName="istio-proxy" containerID="cri-o://75019cdcba71b15789bb57ac75182e113a8368fa207e0db74d381a921bf226c4" gracePeriod=30 Nov 03 03:01:33.127553 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:33.127525 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:33.284357 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:33.284269 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:34.122021 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:34.121988 5463 patch_prober.go:29] interesting pod/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:01:34.122499 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:34.122064 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" podUID="0d32bca4-00c3-4968-92b0-86c8ed0738c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:01:37.533687 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:37.533636 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:38.398526 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.398504 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 03:01:38.435485 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.435453 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:38.439472 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.439447 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-envoy\") pod \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " Nov 03 03:01:38.439589 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.439493 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-data\") pod \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " Nov 03 03:01:38.439589 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.439549 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-token\" (UniqueName: \"kubernetes.io/projected/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-token\") pod \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " Nov 03 03:01:38.439589 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.439585 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-workload-socket\") pod \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " Nov 03 03:01:38.439720 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.439685 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istiod-ca-cert\") pod \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " Nov 03 03:01:38.439988 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.439770 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d32bca4-00c3-4968-92b0-86c8ed0738c9-kube-api-access-zx2mp\" (UniqueName: \"kubernetes.io/projected/0d32bca4-00c3-4968-92b0-86c8ed0738c9-kube-api-access-zx2mp\") pod \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " Nov 03 03:01:38.439988 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.439807 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-data" pod "0d32bca4-00c3-4968-92b0-86c8ed0738c9" (UID: "0d32bca4-00c3-4968-92b0-86c8ed0738c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:01:38.439988 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.439862 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-credential-socket\") pod \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " Nov 03 03:01:38.439988 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.439977 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-workload-socket" pod "0d32bca4-00c3-4968-92b0-86c8ed0738c9" (UID: "0d32bca4-00c3-4968-92b0-86c8ed0738c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:01:38.440303 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.439984 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-podinfo\") pod \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " Nov 03 03:01:38.440303 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.440070 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-workload-certs\") pod \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\" (UID: \"0d32bca4-00c3-4968-92b0-86c8ed0738c9\") " Nov 03 03:01:38.440303 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.440161 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-credential-socket" pod "0d32bca4-00c3-4968-92b0-86c8ed0738c9" (UID: "0d32bca4-00c3-4968-92b0-86c8ed0738c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:01:38.440466 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.440373 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istiod-ca-cert" pod "0d32bca4-00c3-4968-92b0-86c8ed0738c9" (UID: "0d32bca4-00c3-4968-92b0-86c8ed0738c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:01:38.440675 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.440552 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-workload-certs" pod "0d32bca4-00c3-4968-92b0-86c8ed0738c9" (UID: "0d32bca4-00c3-4968-92b0-86c8ed0738c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:01:38.440848 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.440808 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:38.440848 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.440834 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:38.440979 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.440850 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:38.440979 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.440865 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:38.440979 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.440879 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:38.442924 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.442883 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-podinfo" pod "0d32bca4-00c3-4968-92b0-86c8ed0738c9" (UID: "0d32bca4-00c3-4968-92b0-86c8ed0738c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:01:38.442924 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.442892 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-envoy" pod "0d32bca4-00c3-4968-92b0-86c8ed0738c9" (UID: "0d32bca4-00c3-4968-92b0-86c8ed0738c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:01:38.442924 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.442913 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-token" pod "0d32bca4-00c3-4968-92b0-86c8ed0738c9" (UID: "0d32bca4-00c3-4968-92b0-86c8ed0738c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:01:38.443228 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.442935 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d32bca4-00c3-4968-92b0-86c8ed0738c9-kube-api-access-zx2mp" pod "0d32bca4-00c3-4968-92b0-86c8ed0738c9" (UID: "0d32bca4-00c3-4968-92b0-86c8ed0738c9"). InnerVolumeSpecName "kube-api-access-zx2mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:01:38.542168 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.542133 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:38.542168 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.542171 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx2mp\" (UniqueName: \"kubernetes.io/projected/0d32bca4-00c3-4968-92b0-86c8ed0738c9-kube-api-access-zx2mp\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:38.542579 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.542187 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:38.542579 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.542247 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d32bca4-00c3-4968-92b0-86c8ed0738c9-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:01:38.736320 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.736290 5463 generic.go:356] "Generic (PLEG): container finished" podID="0d32bca4-00c3-4968-92b0-86c8ed0738c9" containerID="75019cdcba71b15789bb57ac75182e113a8368fa207e0db74d381a921bf226c4" exitCode=0 Nov 03 03:01:38.736503 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.736338 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" Nov 03 03:01:38.736503 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.736364 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" event={"ID":"0d32bca4-00c3-4968-92b0-86c8ed0738c9","Type":"ContainerDied","Data":"75019cdcba71b15789bb57ac75182e113a8368fa207e0db74d381a921bf226c4"} Nov 03 03:01:38.736503 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.736407 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd" event={"ID":"0d32bca4-00c3-4968-92b0-86c8ed0738c9","Type":"ContainerDied","Data":"89e1f9caa9a9698451add4221564f371de56bd042108a8025b91ccb74425fca0"} Nov 03 03:01:38.736503 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.736428 5463 scope.go:122] "RemoveContainer" containerID="75019cdcba71b15789bb57ac75182e113a8368fa207e0db74d381a921bf226c4" Nov 03 03:01:38.746239 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.746220 5463 scope.go:122] "RemoveContainer" containerID="6519998f7c7b3968fb6b5eba6273f26ae8d70dc57813b4635673e6e1a271e40d" Nov 03 03:01:38.757295 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.757277 5463 scope.go:122] "RemoveContainer" containerID="75019cdcba71b15789bb57ac75182e113a8368fa207e0db74d381a921bf226c4" Nov 03 03:01:38.757562 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:01:38.757544 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75019cdcba71b15789bb57ac75182e113a8368fa207e0db74d381a921bf226c4\": container with ID starting with 75019cdcba71b15789bb57ac75182e113a8368fa207e0db74d381a921bf226c4 not found: ID does not exist" containerID="75019cdcba71b15789bb57ac75182e113a8368fa207e0db74d381a921bf226c4" Nov 03 03:01:38.757629 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.757570 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75019cdcba71b15789bb57ac75182e113a8368fa207e0db74d381a921bf226c4"} err="failed to get container status \"75019cdcba71b15789bb57ac75182e113a8368fa207e0db74d381a921bf226c4\": rpc error: code = NotFound desc = could not find container \"75019cdcba71b15789bb57ac75182e113a8368fa207e0db74d381a921bf226c4\": container with ID starting with 75019cdcba71b15789bb57ac75182e113a8368fa207e0db74d381a921bf226c4 not found: ID does not exist" Nov 03 03:01:38.757629 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.757587 5463 scope.go:122] "RemoveContainer" containerID="6519998f7c7b3968fb6b5eba6273f26ae8d70dc57813b4635673e6e1a271e40d" Nov 03 03:01:38.757862 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:01:38.757844 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6519998f7c7b3968fb6b5eba6273f26ae8d70dc57813b4635673e6e1a271e40d\": container with ID starting with 6519998f7c7b3968fb6b5eba6273f26ae8d70dc57813b4635673e6e1a271e40d not found: ID does not exist" containerID="6519998f7c7b3968fb6b5eba6273f26ae8d70dc57813b4635673e6e1a271e40d" Nov 03 03:01:38.757921 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.757868 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6519998f7c7b3968fb6b5eba6273f26ae8d70dc57813b4635673e6e1a271e40d"} err="failed to get container status \"6519998f7c7b3968fb6b5eba6273f26ae8d70dc57813b4635673e6e1a271e40d\": rpc error: code = NotFound desc = could not find container \"6519998f7c7b3968fb6b5eba6273f26ae8d70dc57813b4635673e6e1a271e40d\": container with ID starting with 6519998f7c7b3968fb6b5eba6273f26ae8d70dc57813b4635673e6e1a271e40d not found: ID does not exist" Nov 03 03:01:38.905766 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.905732 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd"] Nov 03 03:01:38.910517 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:38.910494 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9a5e31-openshift-default-5dc5d749d9-74njd"] Nov 03 03:01:40.443509 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:40.443478 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d32bca4-00c3-4968-92b0-86c8ed0738c9" path="/var/lib/kubelet/pods/0d32bca4-00c3-4968-92b0-86c8ed0738c9/volumes" Nov 03 03:01:43.009588 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:01:43.009555 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:01:43.344034 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:43.343954 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:48.170429 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:48.170393 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:53.058138 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:53.058092 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:57.843667 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:01:57.843629 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:02.538846 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:02.538806 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:03.208253 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:03.208222 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 03:02:03.214928 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:03.214907 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 03:02:07.428806 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:07.428772 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:12.258340 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:12.258303 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:13.009372 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:02:13.009330 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:02:17.025560 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:17.025511 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:21.745779 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:21.745740 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:26.330669 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:26.330632 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:26.484407 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:26.484375 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:31.111508 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:31.111477 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:32.089919 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:32.089883 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:35.848222 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:35.848153 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:39.240767 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:39.240683 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:40.669982 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:40.669940 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:43.008235 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:02:43.008170 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:02:44.918444 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:44.918407 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:45.445823 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:45.445790 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:46.665015 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:46.664978 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:46.851728 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:46.851694 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:47.578827 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:47.578779 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:49.905695 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:49.905661 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:50.434439 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:50.434406 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:55.225842 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:02:55.225813 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:00.018677 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:03:00.018644 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:00.852932 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:03:00.852896 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:04.906126 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:03:04.906089 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:09.586539 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:03:09.586480 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:13.006262 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:03:13.006224 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:03:13.791240 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:03:13.791185 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:14.365746 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:03:14.365687 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:19.040321 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:03:19.040286 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:19.510715 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:03:19.510675 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:23.771212 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:03:23.771142 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:26.608122 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:03:26.608085 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:28.508457 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:03:28.508413 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:33.248049 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:03:33.248009 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:37.919086 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:03:37.919050 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:42.650623 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:03:42.650566 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:43.007719 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:03:43.007643 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:03:47.464002 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:03:47.463956 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:52.218960 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:03:52.218919 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:57.016557 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:03:57.016513 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:58.626582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:03:58.626543 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:58.809024 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:03:58.808988 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:01.767869 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:01.767838 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:06.519385 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:06.519348 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:09.950597 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:09.950495 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:10.589662 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:10.589613 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:10.745104 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:10.745067 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:10.919115 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:10.919079 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:11.333864 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:11.333777 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:13.007907 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:04:13.007872 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:04:14.618752 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:14.618719 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:14.799125 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:14.799087 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:14.967504 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:14.967471 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:16.455241 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:16.455179 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:17.098037 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:17.098003 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:17.931052 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:17.931017 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:18.113110 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.113075 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:18.331355 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.331272 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n"] Nov 03 03:04:18.331983 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.331963 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d7ddd193-7cb2-401b-93b9-407faf6ed1fc" containerName="cleanup" Nov 03 03:04:18.332102 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.331985 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7ddd193-7cb2-401b-93b9-407faf6ed1fc" containerName="cleanup" Nov 03 03:04:18.332102 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332008 5463 cpu_manager.go:414] "containerMap: removing container" podUID="0903eaf9-cfca-4e03-8200-561cff5ab89f" containerName="cleanup" Nov 03 03:04:18.332102 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332015 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0903eaf9-cfca-4e03-8200-561cff5ab89f" containerName="cleanup" Nov 03 03:04:18.332102 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332030 5463 cpu_manager.go:414] "containerMap: removing container" podUID="bc8f16d1-4d9c-4c4b-8449-689172ef80be" containerName="istio-proxy" Nov 03 03:04:18.332102 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332038 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc8f16d1-4d9c-4c4b-8449-689172ef80be" containerName="istio-proxy" Nov 03 03:04:18.332102 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332054 5463 cpu_manager.go:414] "containerMap: removing container" podUID="0d32bca4-00c3-4968-92b0-86c8ed0738c9" containerName="istio-proxy" Nov 03 03:04:18.332102 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332061 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d32bca4-00c3-4968-92b0-86c8ed0738c9" containerName="istio-proxy" Nov 03 03:04:18.332102 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332072 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a6e57752-f49f-4fda-a9f7-8f902dfd6fb2" containerName="cleanup" Nov 03 03:04:18.332102 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332078 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e57752-f49f-4fda-a9f7-8f902dfd6fb2" containerName="cleanup" Nov 03 03:04:18.332102 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332093 5463 cpu_manager.go:414] "containerMap: removing container" podUID="3b38f888-2173-4d9b-a4cb-047d5086e894" containerName="istio-proxy" Nov 03 03:04:18.332102 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332100 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b38f888-2173-4d9b-a4cb-047d5086e894" containerName="istio-proxy" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332113 5463 cpu_manager.go:414] "containerMap: removing container" podUID="ae465e23-7552-4847-8526-028628c3cb05" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332120 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae465e23-7552-4847-8526-028628c3cb05" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332132 5463 cpu_manager.go:414] "containerMap: removing container" podUID="7aa45c20-0b65-4808-80e2-ba8c767966f9" containerName="istio-proxy" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332139 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa45c20-0b65-4808-80e2-ba8c767966f9" containerName="istio-proxy" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332152 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a6e57752-f49f-4fda-a9f7-8f902dfd6fb2" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332158 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e57752-f49f-4fda-a9f7-8f902dfd6fb2" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332170 5463 cpu_manager.go:414] "containerMap: removing container" podUID="ae465e23-7552-4847-8526-028628c3cb05" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332176 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae465e23-7552-4847-8526-028628c3cb05" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332186 5463 cpu_manager.go:414] "containerMap: removing container" podUID="29bcbc40-df9d-489e-9a7d-50f2c49c8312" containerName="istio-proxy" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332209 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29bcbc40-df9d-489e-9a7d-50f2c49c8312" containerName="istio-proxy" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332218 5463 cpu_manager.go:414] "containerMap: removing container" podUID="510e94f4-1d02-4c4b-8928-6253787c69e4" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332225 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510e94f4-1d02-4c4b-8928-6253787c69e4" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332237 5463 cpu_manager.go:414] "containerMap: removing container" podUID="c2475661-edad-4611-b2bc-5930241596fa" containerName="maas-api" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332244 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2475661-edad-4611-b2bc-5930241596fa" containerName="maas-api" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332253 5463 cpu_manager.go:414] "containerMap: removing container" podUID="0903eaf9-cfca-4e03-8200-561cff5ab89f" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332259 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0903eaf9-cfca-4e03-8200-561cff5ab89f" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332276 5463 cpu_manager.go:414] "containerMap: removing container" podUID="3b38f888-2173-4d9b-a4cb-047d5086e894" containerName="istio-proxy" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332283 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b38f888-2173-4d9b-a4cb-047d5086e894" containerName="istio-proxy" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332299 5463 cpu_manager.go:414] "containerMap: removing container" podUID="bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e" containerName="maas-api" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332306 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e" containerName="maas-api" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332318 5463 cpu_manager.go:414] "containerMap: removing container" podUID="0903eaf9-cfca-4e03-8200-561cff5ab89f" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332326 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0903eaf9-cfca-4e03-8200-561cff5ab89f" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332336 5463 cpu_manager.go:414] "containerMap: removing container" podUID="f1300f44-ac64-49c0-9387-8279a643f646" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332343 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1300f44-ac64-49c0-9387-8279a643f646" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332367 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d7ddd193-7cb2-401b-93b9-407faf6ed1fc" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332374 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7ddd193-7cb2-401b-93b9-407faf6ed1fc" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332396 5463 cpu_manager.go:414] "containerMap: removing container" podUID="bc8f16d1-4d9c-4c4b-8449-689172ef80be" containerName="istio-proxy" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332404 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc8f16d1-4d9c-4c4b-8449-689172ef80be" containerName="istio-proxy" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332413 5463 cpu_manager.go:414] "containerMap: removing container" podUID="510e94f4-1d02-4c4b-8928-6253787c69e4" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332420 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510e94f4-1d02-4c4b-8928-6253787c69e4" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332433 5463 cpu_manager.go:414] "containerMap: removing container" podUID="f1300f44-ac64-49c0-9387-8279a643f646" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332440 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1300f44-ac64-49c0-9387-8279a643f646" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332450 5463 cpu_manager.go:414] "containerMap: removing container" podUID="510e94f4-1d02-4c4b-8928-6253787c69e4" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332458 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510e94f4-1d02-4c4b-8928-6253787c69e4" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332615 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b38f888-2173-4d9b-a4cb-047d5086e894" containerName="istio-proxy" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332629 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae465e23-7552-4847-8526-028628c3cb05" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332645 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa45c20-0b65-4808-80e2-ba8c767966f9" containerName="istio-proxy" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332656 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e57752-f49f-4fda-a9f7-8f902dfd6fb2" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332667 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae465e23-7552-4847-8526-028628c3cb05" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332676 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="29bcbc40-df9d-489e-9a7d-50f2c49c8312" containerName="istio-proxy" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332686 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="510e94f4-1d02-4c4b-8928-6253787c69e4" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332695 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2475661-edad-4611-b2bc-5930241596fa" containerName="maas-api" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332703 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0903eaf9-cfca-4e03-8200-561cff5ab89f" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332713 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b38f888-2173-4d9b-a4cb-047d5086e894" containerName="istio-proxy" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332722 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8ec8c4-a60c-48e3-8e8c-9ff53af84a5e" containerName="maas-api" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332732 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0903eaf9-cfca-4e03-8200-561cff5ab89f" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332739 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1300f44-ac64-49c0-9387-8279a643f646" containerName="cleanup" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332748 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d32bca4-00c3-4968-92b0-86c8ed0738c9" containerName="istio-proxy" Nov 03 03:04:18.332735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332759 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7ddd193-7cb2-401b-93b9-407faf6ed1fc" containerName="cleanup" Nov 03 03:04:18.335213 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332767 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7ddd193-7cb2-401b-93b9-407faf6ed1fc" containerName="cleanup" Nov 03 03:04:18.335213 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332777 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc8f16d1-4d9c-4c4b-8449-689172ef80be" containerName="istio-proxy" Nov 03 03:04:18.335213 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332786 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="510e94f4-1d02-4c4b-8928-6253787c69e4" containerName="cleanup" Nov 03 03:04:18.335213 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332794 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1300f44-ac64-49c0-9387-8279a643f646" containerName="cleanup" Nov 03 03:04:18.335213 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332802 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1300f44-ac64-49c0-9387-8279a643f646" containerName="cleanup" Nov 03 03:04:18.335213 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332816 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc8f16d1-4d9c-4c4b-8449-689172ef80be" containerName="istio-proxy" Nov 03 03:04:18.335213 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.332827 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e57752-f49f-4fda-a9f7-8f902dfd6fb2" containerName="cleanup" Nov 03 03:04:18.336828 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.336807 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n" Nov 03 03:04:18.386974 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.386937 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n"] Nov 03 03:04:18.432215 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.428988 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ce4448-0768-4a3c-9f54-a0a377e944bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n\" (UID: \"18ce4448-0768-4a3c-9f54-a0a377e944bf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n" Nov 03 03:04:18.432215 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.429182 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcbtj\" (UniqueName: \"kubernetes.io/projected/18ce4448-0768-4a3c-9f54-a0a377e944bf-kube-api-access-qcbtj\") pod \"payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n\" (UID: \"18ce4448-0768-4a3c-9f54-a0a377e944bf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n" Nov 03 03:04:18.434416 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.434390 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-85e72378\"" Nov 03 03:04:18.443223 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.435279 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-85e72378-dockercfg-c59rg\"" Nov 03 03:04:18.510386 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.510358 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw"] Nov 03 03:04:18.511034 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.511020 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d7ddd193-7cb2-401b-93b9-407faf6ed1fc" containerName="cleanup" Nov 03 03:04:18.511086 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.511046 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7ddd193-7cb2-401b-93b9-407faf6ed1fc" containerName="cleanup" Nov 03 03:04:18.511124 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.511098 5463 cpu_manager.go:414] "containerMap: removing container" podUID="0d32bca4-00c3-4968-92b0-86c8ed0738c9" containerName="istio-proxy" Nov 03 03:04:18.511124 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.511106 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d32bca4-00c3-4968-92b0-86c8ed0738c9" containerName="istio-proxy" Nov 03 03:04:18.511124 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.511118 5463 cpu_manager.go:414] "containerMap: removing container" podUID="ae465e23-7552-4847-8526-028628c3cb05" containerName="cleanup" Nov 03 03:04:18.511124 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.511123 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae465e23-7552-4847-8526-028628c3cb05" containerName="cleanup" Nov 03 03:04:18.511283 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.511135 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a6e57752-f49f-4fda-a9f7-8f902dfd6fb2" containerName="cleanup" Nov 03 03:04:18.511283 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.511140 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e57752-f49f-4fda-a9f7-8f902dfd6fb2" containerName="cleanup" Nov 03 03:04:18.511283 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.511163 5463 cpu_manager.go:414] "containerMap: removing container" podUID="f1300f44-ac64-49c0-9387-8279a643f646" containerName="cleanup" Nov 03 03:04:18.511283 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.511169 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1300f44-ac64-49c0-9387-8279a643f646" containerName="cleanup" Nov 03 03:04:18.511404 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.511293 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e57752-f49f-4fda-a9f7-8f902dfd6fb2" containerName="cleanup" Nov 03 03:04:18.511404 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.511308 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae465e23-7552-4847-8526-028628c3cb05" containerName="cleanup" Nov 03 03:04:18.511404 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.511317 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0903eaf9-cfca-4e03-8200-561cff5ab89f" containerName="cleanup" Nov 03 03:04:18.511404 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.511331 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d32bca4-00c3-4968-92b0-86c8ed0738c9" containerName="istio-proxy" Nov 03 03:04:18.511404 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.511339 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7ddd193-7cb2-401b-93b9-407faf6ed1fc" containerName="cleanup" Nov 03 03:04:18.511404 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.511348 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="510e94f4-1d02-4c4b-8928-6253787c69e4" containerName="cleanup" Nov 03 03:04:18.514871 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.514850 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw" Nov 03 03:04:18.530179 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.530152 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcbtj\" (UniqueName: \"kubernetes.io/projected/18ce4448-0768-4a3c-9f54-a0a377e944bf-kube-api-access-qcbtj\") pod \"payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n\" (UID: \"18ce4448-0768-4a3c-9f54-a0a377e944bf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n" Nov 03 03:04:18.530364 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.530341 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ce4448-0768-4a3c-9f54-a0a377e944bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n\" (UID: \"18ce4448-0768-4a3c-9f54-a0a377e944bf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n" Nov 03 03:04:18.531333 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.531308 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ce4448-0768-4a3c-9f54-a0a377e944bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n\" (UID: \"18ce4448-0768-4a3c-9f54-a0a377e944bf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n" Nov 03 03:04:18.532736 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.532717 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw"] Nov 03 03:04:18.588837 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.588764 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcbtj\" (UniqueName: \"kubernetes.io/projected/18ce4448-0768-4a3c-9f54-a0a377e944bf-kube-api-access-qcbtj\") pod \"payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n\" (UID: \"18ce4448-0768-4a3c-9f54-a0a377e944bf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n" Nov 03 03:04:18.631237 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.631171 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/593ad14d-f2d5-4cc8-96fb-41e6f1253b72-plugins-config-volume\") pod \"payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw\" (UID: \"593ad14d-f2d5-4cc8-96fb-41e6f1253b72\") " pod="openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw" Nov 03 03:04:18.631569 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.631542 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twcx9\" (UniqueName: \"kubernetes.io/projected/593ad14d-f2d5-4cc8-96fb-41e6f1253b72-kube-api-access-twcx9\") pod \"payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw\" (UID: \"593ad14d-f2d5-4cc8-96fb-41e6f1253b72\") " pod="openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw" Nov 03 03:04:18.652823 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.652794 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n" Nov 03 03:04:18.733122 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.733091 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/593ad14d-f2d5-4cc8-96fb-41e6f1253b72-plugins-config-volume\") pod \"payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw\" (UID: \"593ad14d-f2d5-4cc8-96fb-41e6f1253b72\") " pod="openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw" Nov 03 03:04:18.733417 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.733396 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twcx9\" (UniqueName: \"kubernetes.io/projected/593ad14d-f2d5-4cc8-96fb-41e6f1253b72-kube-api-access-twcx9\") pod \"payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw\" (UID: \"593ad14d-f2d5-4cc8-96fb-41e6f1253b72\") " pod="openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw" Nov 03 03:04:18.735391 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.735366 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/593ad14d-f2d5-4cc8-96fb-41e6f1253b72-plugins-config-volume\") pod \"payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw\" (UID: \"593ad14d-f2d5-4cc8-96fb-41e6f1253b72\") " pod="openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw" Nov 03 03:04:18.793478 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.793441 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twcx9\" (UniqueName: \"kubernetes.io/projected/593ad14d-f2d5-4cc8-96fb-41e6f1253b72-kube-api-access-twcx9\") pod \"payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw\" (UID: \"593ad14d-f2d5-4cc8-96fb-41e6f1253b72\") " pod="openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw" Nov 03 03:04:18.835485 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:18.835457 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw" Nov 03 03:04:19.044297 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:04:19.044260 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ce4448_0768_4a3c_9f54_a0a377e944bf.slice/crio-c1712908acc82164f9b33cc23f738a0d5cef404300b15145a9b4299907a7609a WatchSource:0}: Error finding container c1712908acc82164f9b33cc23f738a0d5cef404300b15145a9b4299907a7609a: Status 404 returned error can't find the container with id c1712908acc82164f9b33cc23f738a0d5cef404300b15145a9b4299907a7609a Nov 03 03:04:19.047404 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:19.047376 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n"] Nov 03 03:04:19.226139 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:04:19.226107 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593ad14d_f2d5_4cc8_96fb_41e6f1253b72.slice/crio-8ae8b405e209c94227b23ff33751152bd28817e15e9b81e68d14fdf3d5188b25 WatchSource:0}: Error finding container 8ae8b405e209c94227b23ff33751152bd28817e15e9b81e68d14fdf3d5188b25: Status 404 returned error can't find the container with id 8ae8b405e209c94227b23ff33751152bd28817e15e9b81e68d14fdf3d5188b25 Nov 03 03:04:19.228641 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:19.228619 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw"] Nov 03 03:04:20.006400 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:20.006359 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw" event={"ID":"593ad14d-f2d5-4cc8-96fb-41e6f1253b72","Type":"ContainerStarted","Data":"f8fdc7c7e76ac57327c4821c2aba0b49d314b92c8c4ec8c2b21ea0c34c948d3e"} Nov 03 03:04:20.006592 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:20.006430 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw" event={"ID":"593ad14d-f2d5-4cc8-96fb-41e6f1253b72","Type":"ContainerStarted","Data":"8ae8b405e209c94227b23ff33751152bd28817e15e9b81e68d14fdf3d5188b25"} Nov 03 03:04:20.006592 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:20.006583 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw" Nov 03 03:04:20.008349 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:20.008327 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n" event={"ID":"18ce4448-0768-4a3c-9f54-a0a377e944bf","Type":"ContainerStarted","Data":"19cc3c154223e62809d13879611a241d48b394040fda141285fb99509e0c1346"} Nov 03 03:04:20.008482 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:20.008366 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n" event={"ID":"18ce4448-0768-4a3c-9f54-a0a377e944bf","Type":"ContainerStarted","Data":"c1712908acc82164f9b33cc23f738a0d5cef404300b15145a9b4299907a7609a"} Nov 03 03:04:20.008482 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:20.008397 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n" Nov 03 03:04:20.123716 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:20.123665 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw" podStartSLOduration=2.123649048 podStartE2EDuration="2.123649048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:04:20.11988623 +0000 UTC m=+4341.803995385" watchObservedRunningTime="2026-11-03 03:04:20.123649048 +0000 UTC m=+4341.807758205" Nov 03 03:04:20.238370 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:20.238319 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n" podStartSLOduration=2.238304791 podStartE2EDuration="2.238304791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:04:20.234103778 +0000 UTC m=+4341.918212946" watchObservedRunningTime="2026-11-03 03:04:20.238304791 +0000 UTC m=+4341.922413973" Nov 03 03:04:21.515791 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:21.515762 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:26.316817 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:26.316782 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:26.919537 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:26.919500 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:28.883504 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:28.883472 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:31.018423 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:31.018395 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw" Nov 03 03:04:31.018808 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:31.018467 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n" Nov 03 03:04:31.113509 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:31.113478 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:35.858622 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:35.858584 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:40.499879 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:40.499848 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:40.646464 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:40.646431 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:43.012239 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:04:43.012172 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:04:45.302345 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:45.302296 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:50.028215 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:50.028096 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:54.229864 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:54.229833 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:54.815173 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:54.815131 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:57.134397 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:57.134367 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:59.516520 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:04:59.516477 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:04.225153 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:04.225118 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:09.047718 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:09.047680 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:13.007524 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:05:13.007485 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:05:13.818472 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:13.818434 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:18.524362 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:18.524327 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:22.982290 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:22.982253 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm"] Nov 03 03:05:22.986942 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:22.986919 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" Nov 03 03:05:23.039128 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:23.039101 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-85e72378-serving-cert\"" Nov 03 03:05:23.040177 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:23.040150 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm"] Nov 03 03:05:23.074669 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:23.074637 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-maas-api-tls\") pod \"maas-api-e2e-mt-85e72378-6c557d985d-7v8pm\" (UID: \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" Nov 03 03:05:23.074842 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:23.074779 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-openshift-service-ca\") pod \"maas-api-e2e-mt-85e72378-6c557d985d-7v8pm\" (UID: \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" Nov 03 03:05:23.074922 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:23.074886 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-894hh\" (UniqueName: \"kubernetes.io/projected/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-kube-api-access-894hh\") pod \"maas-api-e2e-mt-85e72378-6c557d985d-7v8pm\" (UID: \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" Nov 03 03:05:23.075019 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:23.074997 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-metrics-certs\") pod \"maas-api-e2e-mt-85e72378-6c557d985d-7v8pm\" (UID: \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" Nov 03 03:05:23.176602 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:23.176569 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-894hh\" (UniqueName: \"kubernetes.io/projected/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-kube-api-access-894hh\") pod \"maas-api-e2e-mt-85e72378-6c557d985d-7v8pm\" (UID: \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" Nov 03 03:05:23.176778 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:23.176642 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-metrics-certs\") pod \"maas-api-e2e-mt-85e72378-6c557d985d-7v8pm\" (UID: \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" Nov 03 03:05:23.176778 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:23.176734 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-maas-api-tls\") pod \"maas-api-e2e-mt-85e72378-6c557d985d-7v8pm\" (UID: \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" Nov 03 03:05:23.176905 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:23.176779 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-openshift-service-ca\") pod \"maas-api-e2e-mt-85e72378-6c557d985d-7v8pm\" (UID: \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" Nov 03 03:05:23.176905 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:05:23.176886 5463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-serving-cert: secret "maas-api-e2e-mt-85e72378-serving-cert" not found Nov 03 03:05:23.177025 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:05:23.176999 5463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-maas-api-tls podName:e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c nodeName:}" failed. No retries permitted until 2026-11-03 03:05:23.676975696 +0000 UTC m=+4405.361084851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-maas-api-tls") pod "maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" (UID: "e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c") : secret "maas-api-e2e-mt-85e72378-serving-cert" not found Nov 03 03:05:23.177514 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:23.177492 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-openshift-service-ca\") pod \"maas-api-e2e-mt-85e72378-6c557d985d-7v8pm\" (UID: \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" Nov 03 03:05:23.179255 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:23.179233 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-metrics-certs\") pod \"maas-api-e2e-mt-85e72378-6c557d985d-7v8pm\" (UID: \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" Nov 03 03:05:23.193549 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:23.193525 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:23.235874 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:23.235772 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-894hh\" (UniqueName: \"kubernetes.io/projected/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-kube-api-access-894hh\") pod \"maas-api-e2e-mt-85e72378-6c557d985d-7v8pm\" (UID: \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" Nov 03 03:05:23.369097 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:23.369047 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:23.522944 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:23.522849 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:23.684657 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:23.684623 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-maas-api-tls\") pod \"maas-api-e2e-mt-85e72378-6c557d985d-7v8pm\" (UID: \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" Nov 03 03:05:23.687051 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:23.687027 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-maas-api-tls\") pod \"maas-api-e2e-mt-85e72378-6c557d985d-7v8pm\" (UID: \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" Nov 03 03:05:23.902674 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:23.902648 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" Nov 03 03:05:24.278153 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:05:24.278104 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c681fe_6b83_4e9e_a3ae_29b032bb7c1c.slice/crio-c4f617dae9e8097498bb1b31d225f4891e637c88131529610eb9a1b2277a37fd WatchSource:0}: Error finding container c4f617dae9e8097498bb1b31d225f4891e637c88131529610eb9a1b2277a37fd: Status 404 returned error can't find the container with id c4f617dae9e8097498bb1b31d225f4891e637c88131529610eb9a1b2277a37fd Nov 03 03:05:24.279679 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:24.279657 5463 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:05:24.282235 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:24.282207 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm"] Nov 03 03:05:24.529726 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:24.529649 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" event={"ID":"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c","Type":"ContainerStarted","Data":"c4f617dae9e8097498bb1b31d225f4891e637c88131529610eb9a1b2277a37fd"} Nov 03 03:05:25.538203 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:25.538151 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" event={"ID":"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c","Type":"ContainerStarted","Data":"e4ca537a65d442533402cd7eb1a02e06c05ddb1664bb93371ef01e8fec1899e5"} Nov 03 03:05:25.538600 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:25.538359 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" Nov 03 03:05:25.655097 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:25.655049 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" podStartSLOduration=2.632278158 podStartE2EDuration="3.655036516s" totalImagesPullingTime="1.022758358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:05:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:05:25.654893269 +0000 UTC m=+4407.339002423" watchObservedRunningTime="2026-11-03 03:05:25.655036516 +0000 UTC m=+4407.339145672" Nov 03 03:05:27.241058 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:27.241024 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:28.160223 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:28.160174 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:31.550030 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:31.550000 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" Nov 03 03:05:32.932451 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:32.932417 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:34.884584 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:34.884535 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:37.716350 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:37.716314 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:41.278833 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:41.278745 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:41.500290 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:41.500251 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:41.765789 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:41.765761 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:42.516646 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:42.516609 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:43.006433 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:05:43.006385 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:05:47.363805 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:47.363759 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:47.537496 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:47.537462 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:47.971421 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:47.971387 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:52.035565 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:52.035523 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:53.928494 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:53.928460 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:55.142282 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:55.142247 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl"] Nov 03 03:05:55.408814 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:55.408779 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:55.609456 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:55.609427 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:56.054307 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:56.054274 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:57.014045 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:57.014014 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:59.500809 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:05:59.500779 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:00.143236 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:00.143144 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl" podUID="f4cd396d-6d5a-4b80-ad42-48646deb9e26" containerName="payload-pre-processing" containerID="cri-o://80043db282d2ade8e2e834587f37e3569ccd2281d3eeab54218404163956033e" gracePeriod=25 Nov 03 03:06:00.749642 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:00.749604 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl" podUID="f4cd396d-6d5a-4b80-ad42-48646deb9e26" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:06:02.102648 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:02.102610 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:07.012287 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:07.012245 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:10.749558 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:10.749515 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl" podUID="f4cd396d-6d5a-4b80-ad42-48646deb9e26" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:06:11.718759 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:11.718718 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:13.006621 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:06:13.006580 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:06:16.464907 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:16.464858 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:20.432649 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:20.432624 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl" Nov 03 03:06:20.440772 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:20.440745 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4cd396d-6d5a-4b80-ad42-48646deb9e26-kube-api-access-8mxlj\" (UniqueName: \"kubernetes.io/projected/f4cd396d-6d5a-4b80-ad42-48646deb9e26-kube-api-access-8mxlj\") pod \"f4cd396d-6d5a-4b80-ad42-48646deb9e26\" (UID: \"f4cd396d-6d5a-4b80-ad42-48646deb9e26\") " Nov 03 03:06:20.441028 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:20.441012 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4cd396d-6d5a-4b80-ad42-48646deb9e26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4cd396d-6d5a-4b80-ad42-48646deb9e26-plugins-config-volume\") pod \"f4cd396d-6d5a-4b80-ad42-48646deb9e26\" (UID: \"f4cd396d-6d5a-4b80-ad42-48646deb9e26\") " Nov 03 03:06:20.441355 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:20.441335 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4cd396d-6d5a-4b80-ad42-48646deb9e26-plugins-config-volume" pod "f4cd396d-6d5a-4b80-ad42-48646deb9e26" (UID: "f4cd396d-6d5a-4b80-ad42-48646deb9e26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:06:20.441509 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:20.441492 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4cd396d-6d5a-4b80-ad42-48646deb9e26-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:20.442818 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:20.442796 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4cd396d-6d5a-4b80-ad42-48646deb9e26-kube-api-access-8mxlj" pod "f4cd396d-6d5a-4b80-ad42-48646deb9e26" (UID: "f4cd396d-6d5a-4b80-ad42-48646deb9e26"). InnerVolumeSpecName "kube-api-access-8mxlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:06:20.543482 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:20.543439 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mxlj\" (UniqueName: \"kubernetes.io/projected/f4cd396d-6d5a-4b80-ad42-48646deb9e26-kube-api-access-8mxlj\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:20.978409 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:20.978373 5463 generic.go:356] "Generic (PLEG): container finished" podID="f4cd396d-6d5a-4b80-ad42-48646deb9e26" containerID="80043db282d2ade8e2e834587f37e3569ccd2281d3eeab54218404163956033e" exitCode=0 Nov 03 03:06:20.978604 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:20.978425 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl" Nov 03 03:06:20.978604 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:20.978462 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl" event={"ID":"f4cd396d-6d5a-4b80-ad42-48646deb9e26","Type":"ContainerDied","Data":"80043db282d2ade8e2e834587f37e3569ccd2281d3eeab54218404163956033e"} Nov 03 03:06:20.978604 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:20.978519 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl" event={"ID":"f4cd396d-6d5a-4b80-ad42-48646deb9e26","Type":"ContainerDied","Data":"5415a85dec97c05460bf35c1f464fe90713cbdd228037a5a14e14f042ba940a4"} Nov 03 03:06:20.978604 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:20.978548 5463 scope.go:122] "RemoveContainer" containerID="80043db282d2ade8e2e834587f37e3569ccd2281d3eeab54218404163956033e" Nov 03 03:06:20.991701 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:20.991629 5463 scope.go:122] "RemoveContainer" containerID="80043db282d2ade8e2e834587f37e3569ccd2281d3eeab54218404163956033e" Nov 03 03:06:20.992544 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:06:20.992443 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80043db282d2ade8e2e834587f37e3569ccd2281d3eeab54218404163956033e\": container with ID starting with 80043db282d2ade8e2e834587f37e3569ccd2281d3eeab54218404163956033e not found: ID does not exist" containerID="80043db282d2ade8e2e834587f37e3569ccd2281d3eeab54218404163956033e" Nov 03 03:06:20.992544 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:20.992479 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80043db282d2ade8e2e834587f37e3569ccd2281d3eeab54218404163956033e"} err="failed to get container status \"80043db282d2ade8e2e834587f37e3569ccd2281d3eeab54218404163956033e\": rpc error: code = NotFound desc = could not find container \"80043db282d2ade8e2e834587f37e3569ccd2281d3eeab54218404163956033e\": container with ID starting with 80043db282d2ade8e2e834587f37e3569ccd2281d3eeab54218404163956033e not found: ID does not exist" Nov 03 03:06:21.160532 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:21.160488 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl"] Nov 03 03:06:21.204746 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:21.204714 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-965a07-854559566c-z7tzl"] Nov 03 03:06:21.244874 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:21.244796 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:22.443871 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:22.443842 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4cd396d-6d5a-4b80-ad42-48646deb9e26" path="/var/lib/kubelet/pods/f4cd396d-6d5a-4b80-ad42-48646deb9e26/volumes" Nov 03 03:06:25.942457 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:25.942415 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:27.590663 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:27.590633 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:30.720126 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:30.720084 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:35.421283 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:35.421250 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:40.150110 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:40.150050 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:43.009530 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:06:43.009495 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:06:43.534986 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:43.534951 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:44.484478 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:44.484444 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:44.667511 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:44.667479 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:45.215679 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:45.215636 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:46.954232 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:46.953625 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:49.267089 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:49.267056 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9"] Nov 03 03:06:49.267590 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:49.267321 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" podUID="6f084fa9-ab75-4909-b65b-2011b6472ede" containerName="maas-api" containerID="cri-o://984ed74a227e3fb90e649ac3ec1013707d912c9861112785d041a4e22b749cdf" gracePeriod=30 Nov 03 03:06:49.391036 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:49.390993 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:49.564636 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:49.564608 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" Nov 03 03:06:49.621597 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:49.621564 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f084fa9-ab75-4909-b65b-2011b6472ede-kube-api-access-b7s8f\" (UniqueName: \"kubernetes.io/projected/6f084fa9-ab75-4909-b65b-2011b6472ede-kube-api-access-b7s8f\") pod \"6f084fa9-ab75-4909-b65b-2011b6472ede\" (UID: \"6f084fa9-ab75-4909-b65b-2011b6472ede\") " Nov 03 03:06:49.621753 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:49.621623 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f084fa9-ab75-4909-b65b-2011b6472ede-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f084fa9-ab75-4909-b65b-2011b6472ede-metrics-certs\") pod \"6f084fa9-ab75-4909-b65b-2011b6472ede\" (UID: \"6f084fa9-ab75-4909-b65b-2011b6472ede\") " Nov 03 03:06:49.621753 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:49.621734 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f084fa9-ab75-4909-b65b-2011b6472ede-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f084fa9-ab75-4909-b65b-2011b6472ede-maas-api-tls\") pod \"6f084fa9-ab75-4909-b65b-2011b6472ede\" (UID: \"6f084fa9-ab75-4909-b65b-2011b6472ede\") " Nov 03 03:06:49.621847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:49.621771 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f084fa9-ab75-4909-b65b-2011b6472ede-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f084fa9-ab75-4909-b65b-2011b6472ede-openshift-service-ca\") pod \"6f084fa9-ab75-4909-b65b-2011b6472ede\" (UID: \"6f084fa9-ab75-4909-b65b-2011b6472ede\") " Nov 03 03:06:49.622252 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:49.622222 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f084fa9-ab75-4909-b65b-2011b6472ede-openshift-service-ca" pod "6f084fa9-ab75-4909-b65b-2011b6472ede" (UID: "6f084fa9-ab75-4909-b65b-2011b6472ede"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:06:49.623749 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:49.623719 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f084fa9-ab75-4909-b65b-2011b6472ede-maas-api-tls" pod "6f084fa9-ab75-4909-b65b-2011b6472ede" (UID: "6f084fa9-ab75-4909-b65b-2011b6472ede"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:06:49.623875 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:49.623754 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f084fa9-ab75-4909-b65b-2011b6472ede-metrics-certs" pod "6f084fa9-ab75-4909-b65b-2011b6472ede" (UID: "6f084fa9-ab75-4909-b65b-2011b6472ede"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:06:49.623875 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:49.623789 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f084fa9-ab75-4909-b65b-2011b6472ede-kube-api-access-b7s8f" pod "6f084fa9-ab75-4909-b65b-2011b6472ede" (UID: "6f084fa9-ab75-4909-b65b-2011b6472ede"). InnerVolumeSpecName "kube-api-access-b7s8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:06:49.723592 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:49.723541 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7s8f\" (UniqueName: \"kubernetes.io/projected/6f084fa9-ab75-4909-b65b-2011b6472ede-kube-api-access-b7s8f\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:49.723592 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:49.723586 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f084fa9-ab75-4909-b65b-2011b6472ede-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:49.723862 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:49.723603 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f084fa9-ab75-4909-b65b-2011b6472ede-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:49.723862 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:49.723617 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f084fa9-ab75-4909-b65b-2011b6472ede-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:50.234396 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:50.233509 5463 generic.go:356] "Generic (PLEG): container finished" podID="6f084fa9-ab75-4909-b65b-2011b6472ede" containerID="984ed74a227e3fb90e649ac3ec1013707d912c9861112785d041a4e22b749cdf" exitCode=0 Nov 03 03:06:50.234396 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:50.233751 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" event={"ID":"6f084fa9-ab75-4909-b65b-2011b6472ede","Type":"ContainerDied","Data":"984ed74a227e3fb90e649ac3ec1013707d912c9861112785d041a4e22b749cdf"} Nov 03 03:06:50.234396 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:50.233801 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" event={"ID":"6f084fa9-ab75-4909-b65b-2011b6472ede","Type":"ContainerDied","Data":"e0ce8c36b97812260174262004334073b59b49d6fc5245be1fd93fc2f347e607"} Nov 03 03:06:50.234396 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:50.233833 5463 scope.go:122] "RemoveContainer" containerID="984ed74a227e3fb90e649ac3ec1013707d912c9861112785d041a4e22b749cdf" Nov 03 03:06:50.234396 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:50.233990 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9" Nov 03 03:06:50.253381 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:50.253357 5463 scope.go:122] "RemoveContainer" containerID="984ed74a227e3fb90e649ac3ec1013707d912c9861112785d041a4e22b749cdf" Nov 03 03:06:50.253758 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:06:50.253736 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"984ed74a227e3fb90e649ac3ec1013707d912c9861112785d041a4e22b749cdf\": container with ID starting with 984ed74a227e3fb90e649ac3ec1013707d912c9861112785d041a4e22b749cdf not found: ID does not exist" containerID="984ed74a227e3fb90e649ac3ec1013707d912c9861112785d041a4e22b749cdf" Nov 03 03:06:50.253838 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:50.253766 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"984ed74a227e3fb90e649ac3ec1013707d912c9861112785d041a4e22b749cdf"} err="failed to get container status \"984ed74a227e3fb90e649ac3ec1013707d912c9861112785d041a4e22b749cdf\": rpc error: code = NotFound desc = could not find container \"984ed74a227e3fb90e649ac3ec1013707d912c9861112785d041a4e22b749cdf\": container with ID starting with 984ed74a227e3fb90e649ac3ec1013707d912c9861112785d041a4e22b749cdf not found: ID does not exist" Nov 03 03:06:50.400025 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:50.399988 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9"] Nov 03 03:06:50.403040 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:50.403015 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-965a07-7f479b5f78-hfsc9"] Nov 03 03:06:50.444476 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:50.444447 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f084fa9-ab75-4909-b65b-2011b6472ede" path="/var/lib/kubelet/pods/6f084fa9-ab75-4909-b65b-2011b6472ede/volumes" Nov 03 03:06:50.548808 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:50.548671 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:51.245261 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:51.245209 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:51.413509 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:51.413445 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:52.224703 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:52.224670 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:56.026081 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:56.026044 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:57.088854 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.088817 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65"] Nov 03 03:06:57.089513 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.089497 5463 cpu_manager.go:414] "containerMap: removing container" podUID="f4cd396d-6d5a-4b80-ad42-48646deb9e26" containerName="payload-pre-processing" Nov 03 03:06:57.089576 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.089514 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4cd396d-6d5a-4b80-ad42-48646deb9e26" containerName="payload-pre-processing" Nov 03 03:06:57.089576 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.089550 5463 cpu_manager.go:414] "containerMap: removing container" podUID="6f084fa9-ab75-4909-b65b-2011b6472ede" containerName="maas-api" Nov 03 03:06:57.089576 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.089556 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f084fa9-ab75-4909-b65b-2011b6472ede" containerName="maas-api" Nov 03 03:06:57.089680 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.089664 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4cd396d-6d5a-4b80-ad42-48646deb9e26" containerName="payload-pre-processing" Nov 03 03:06:57.089680 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.089678 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f084fa9-ab75-4909-b65b-2011b6472ede" containerName="maas-api" Nov 03 03:06:57.094663 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.094645 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:57.146471 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.146441 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65"] Nov 03 03:06:57.193457 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.193401 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-78b5c5\"/\"openshift-service-ca.crt\"" Nov 03 03:06:57.193457 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.193440 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-78b5c5\"/\"kube-root-ca.crt\"" Nov 03 03:06:57.193656 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.193527 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-78b5c5\"/\"default-dockercfg-cdhld\"" Nov 03 03:06:57.197759 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.197744 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-78b5c5\"/\"sub-test-model-78b5c5-kserve-self-signed-certs\"" Nov 03 03:06:57.204374 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.204352 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:57.225394 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.225369 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtqwd\" (UniqueName: \"kubernetes.io/projected/1ee1477e-0bf0-428b-ac3f-03e5f217116f-kube-api-access-vtqwd\") pod \"sub-test-model-78b5c5-kserve-697d6785c-xgv65\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:57.225500 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.225483 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-dshm\") pod \"sub-test-model-78b5c5-kserve-697d6785c-xgv65\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:57.225556 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.225515 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-model-cache\") pod \"sub-test-model-78b5c5-kserve-697d6785c-xgv65\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:57.225556 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.225542 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-tmp-dir\") pod \"sub-test-model-78b5c5-kserve-697d6785c-xgv65\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:57.225630 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.225568 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-home\") pod \"sub-test-model-78b5c5-kserve-697d6785c-xgv65\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:57.225712 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.225695 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee1477e-0bf0-428b-ac3f-03e5f217116f-tls-certs\") pod \"sub-test-model-78b5c5-kserve-697d6785c-xgv65\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:57.326730 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.326694 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtqwd\" (UniqueName: \"kubernetes.io/projected/1ee1477e-0bf0-428b-ac3f-03e5f217116f-kube-api-access-vtqwd\") pod \"sub-test-model-78b5c5-kserve-697d6785c-xgv65\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:57.326919 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.326792 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-dshm\") pod \"sub-test-model-78b5c5-kserve-697d6785c-xgv65\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:57.326919 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.326812 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-model-cache\") pod \"sub-test-model-78b5c5-kserve-697d6785c-xgv65\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:57.326919 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.326834 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-tmp-dir\") pod \"sub-test-model-78b5c5-kserve-697d6785c-xgv65\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:57.326919 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.326856 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-home\") pod \"sub-test-model-78b5c5-kserve-697d6785c-xgv65\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:57.326919 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.326891 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee1477e-0bf0-428b-ac3f-03e5f217116f-tls-certs\") pod \"sub-test-model-78b5c5-kserve-697d6785c-xgv65\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:57.327304 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.327277 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-model-cache\") pod \"sub-test-model-78b5c5-kserve-697d6785c-xgv65\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:57.327391 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.327346 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-tmp-dir\") pod \"sub-test-model-78b5c5-kserve-697d6785c-xgv65\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:57.327433 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.327382 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-home\") pod \"sub-test-model-78b5c5-kserve-697d6785c-xgv65\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:57.329251 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.329229 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-dshm\") pod \"sub-test-model-78b5c5-kserve-697d6785c-xgv65\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:57.329495 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.329475 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee1477e-0bf0-428b-ac3f-03e5f217116f-tls-certs\") pod \"sub-test-model-78b5c5-kserve-697d6785c-xgv65\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:57.384298 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.384240 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtqwd\" (UniqueName: \"kubernetes.io/projected/1ee1477e-0bf0-428b-ac3f-03e5f217116f-kube-api-access-vtqwd\") pod \"sub-test-model-78b5c5-kserve-697d6785c-xgv65\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:57.408264 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.408239 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:57.786922 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:06:57.786895 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ee1477e_0bf0_428b_ac3f_03e5f217116f.slice/crio-b4ddfc564d315de80d0de3545814b0274a834e475daae5edfd30d297810a86d8 WatchSource:0}: Error finding container b4ddfc564d315de80d0de3545814b0274a834e475daae5edfd30d297810a86d8: Status 404 returned error can't find the container with id b4ddfc564d315de80d0de3545814b0274a834e475daae5edfd30d297810a86d8 Nov 03 03:06:57.790006 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:57.789984 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65"] Nov 03 03:06:58.294809 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:58.294769 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" event={"ID":"1ee1477e-0bf0-428b-ac3f-03e5f217116f","Type":"ContainerStarted","Data":"8bfa16eed80097a3c309f3f43ad7b2c6b18d09d1eb64af541b6f333f94ce6d80"} Nov 03 03:06:58.295232 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:58.294822 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" event={"ID":"1ee1477e-0bf0-428b-ac3f-03e5f217116f","Type":"ContainerStarted","Data":"b4ddfc564d315de80d0de3545814b0274a834e475daae5edfd30d297810a86d8"} Nov 03 03:06:58.408758 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:58.408713 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:06:58.412538 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:58.412494 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" podStartSLOduration=1.4124811369999999 podStartE2EDuration="1.412481137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:06:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:06:58.409374529 +0000 UTC m=+4500.093483729" watchObservedRunningTime="2026-11-03 03:06:58.412481137 +0000 UTC m=+4500.096590292" Nov 03 03:06:58.762482 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:58.762446 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:59.722810 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:59.722768 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:59.901932 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:06:59.901870 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:00.493203 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:00.493166 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw"] Nov 03 03:07:00.602004 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:00.601972 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n"] Nov 03 03:07:00.866920 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:00.866831 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:01.017423 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:01.017389 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:01.294331 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:01.294294 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:01.940616 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:01.940580 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj"] Nov 03 03:07:01.941051 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:01.940824 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" podUID="04066f9c-931a-4c65-a714-57270768f0e5" containerName="istio-proxy" containerID="cri-o://dd45e4e9d6e4d5e77487e402bfedcf23814c8cdfc921a9e1c099a3dcd7e12867" gracePeriod=30 Nov 03 03:07:01.942772 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:01.942742 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:02.798269 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:02.798209 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:03.259883 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:03.259851 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 03:07:03.269572 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:03.269543 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 03:07:04.646020 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:04.645976 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:05.493748 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:05.493708 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw" podUID="593ad14d-f2d5-4cc8-96fb-41e6f1253b72" containerName="payload-processing" containerID="cri-o://f8fdc7c7e76ac57327c4821c2aba0b49d314b92c8c4ec8c2b21ea0c34c948d3e" gracePeriod=25 Nov 03 03:07:05.602999 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:05.602955 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n" podUID="18ce4448-0768-4a3c-9f54-a0a377e944bf" containerName="payload-pre-processing" containerID="cri-o://19cc3c154223e62809d13879611a241d48b394040fda141285fb99509e0c1346" gracePeriod=25 Nov 03 03:07:05.809217 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:05.808934 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw" Nov 03 03:07:05.872378 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:05.872344 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/593ad14d-f2d5-4cc8-96fb-41e6f1253b72-kube-api-access-twcx9\" (UniqueName: \"kubernetes.io/projected/593ad14d-f2d5-4cc8-96fb-41e6f1253b72-kube-api-access-twcx9\") pod \"593ad14d-f2d5-4cc8-96fb-41e6f1253b72\" (UID: \"593ad14d-f2d5-4cc8-96fb-41e6f1253b72\") " Nov 03 03:07:05.872551 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:05.872437 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/593ad14d-f2d5-4cc8-96fb-41e6f1253b72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/593ad14d-f2d5-4cc8-96fb-41e6f1253b72-plugins-config-volume\") pod \"593ad14d-f2d5-4cc8-96fb-41e6f1253b72\" (UID: \"593ad14d-f2d5-4cc8-96fb-41e6f1253b72\") " Nov 03 03:07:05.873076 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:05.872832 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/593ad14d-f2d5-4cc8-96fb-41e6f1253b72-plugins-config-volume" pod "593ad14d-f2d5-4cc8-96fb-41e6f1253b72" (UID: "593ad14d-f2d5-4cc8-96fb-41e6f1253b72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:07:05.873076 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:05.873053 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/593ad14d-f2d5-4cc8-96fb-41e6f1253b72-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:05.875269 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:05.875247 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593ad14d-f2d5-4cc8-96fb-41e6f1253b72-kube-api-access-twcx9" pod "593ad14d-f2d5-4cc8-96fb-41e6f1253b72" (UID: "593ad14d-f2d5-4cc8-96fb-41e6f1253b72"). InnerVolumeSpecName "kube-api-access-twcx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:07:05.903962 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:05.903940 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n" Nov 03 03:07:05.973948 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:05.973908 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ce4448-0768-4a3c-9f54-a0a377e944bf-kube-api-access-qcbtj\" (UniqueName: \"kubernetes.io/projected/18ce4448-0768-4a3c-9f54-a0a377e944bf-kube-api-access-qcbtj\") pod \"18ce4448-0768-4a3c-9f54-a0a377e944bf\" (UID: \"18ce4448-0768-4a3c-9f54-a0a377e944bf\") " Nov 03 03:07:05.974127 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:05.974018 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ce4448-0768-4a3c-9f54-a0a377e944bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ce4448-0768-4a3c-9f54-a0a377e944bf-plugins-config-volume\") pod \"18ce4448-0768-4a3c-9f54-a0a377e944bf\" (UID: \"18ce4448-0768-4a3c-9f54-a0a377e944bf\") " Nov 03 03:07:05.974675 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:05.974644 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twcx9\" (UniqueName: \"kubernetes.io/projected/593ad14d-f2d5-4cc8-96fb-41e6f1253b72-kube-api-access-twcx9\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:05.975060 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:05.975031 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ce4448-0768-4a3c-9f54-a0a377e944bf-plugins-config-volume" pod "18ce4448-0768-4a3c-9f54-a0a377e944bf" (UID: "18ce4448-0768-4a3c-9f54-a0a377e944bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:07:05.977439 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:05.977405 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ce4448-0768-4a3c-9f54-a0a377e944bf-kube-api-access-qcbtj" pod "18ce4448-0768-4a3c-9f54-a0a377e944bf" (UID: "18ce4448-0768-4a3c-9f54-a0a377e944bf"). InnerVolumeSpecName "kube-api-access-qcbtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:07:06.075839 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.075765 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcbtj\" (UniqueName: \"kubernetes.io/projected/18ce4448-0768-4a3c-9f54-a0a377e944bf-kube-api-access-qcbtj\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:06.075839 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.075801 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ce4448-0768-4a3c-9f54-a0a377e944bf-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:06.234939 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.234896 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:06.356760 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.356666 5463 generic.go:356] "Generic (PLEG): container finished" podID="593ad14d-f2d5-4cc8-96fb-41e6f1253b72" containerID="f8fdc7c7e76ac57327c4821c2aba0b49d314b92c8c4ec8c2b21ea0c34c948d3e" exitCode=0 Nov 03 03:07:06.356760 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.356721 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw" Nov 03 03:07:06.357003 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.356752 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw" event={"ID":"593ad14d-f2d5-4cc8-96fb-41e6f1253b72","Type":"ContainerDied","Data":"f8fdc7c7e76ac57327c4821c2aba0b49d314b92c8c4ec8c2b21ea0c34c948d3e"} Nov 03 03:07:06.357003 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.356807 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw" event={"ID":"593ad14d-f2d5-4cc8-96fb-41e6f1253b72","Type":"ContainerDied","Data":"8ae8b405e209c94227b23ff33751152bd28817e15e9b81e68d14fdf3d5188b25"} Nov 03 03:07:06.357003 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.356838 5463 scope.go:122] "RemoveContainer" containerID="f8fdc7c7e76ac57327c4821c2aba0b49d314b92c8c4ec8c2b21ea0c34c948d3e" Nov 03 03:07:06.358794 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.358763 5463 generic.go:356] "Generic (PLEG): container finished" podID="18ce4448-0768-4a3c-9f54-a0a377e944bf" containerID="19cc3c154223e62809d13879611a241d48b394040fda141285fb99509e0c1346" exitCode=0 Nov 03 03:07:06.358951 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.358823 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n" Nov 03 03:07:06.358951 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.358840 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n" event={"ID":"18ce4448-0768-4a3c-9f54-a0a377e944bf","Type":"ContainerDied","Data":"19cc3c154223e62809d13879611a241d48b394040fda141285fb99509e0c1346"} Nov 03 03:07:06.358951 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.358897 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n" event={"ID":"18ce4448-0768-4a3c-9f54-a0a377e944bf","Type":"ContainerDied","Data":"c1712908acc82164f9b33cc23f738a0d5cef404300b15145a9b4299907a7609a"} Nov 03 03:07:06.374733 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.374712 5463 scope.go:122] "RemoveContainer" containerID="f8fdc7c7e76ac57327c4821c2aba0b49d314b92c8c4ec8c2b21ea0c34c948d3e" Nov 03 03:07:06.375080 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:07:06.375059 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8fdc7c7e76ac57327c4821c2aba0b49d314b92c8c4ec8c2b21ea0c34c948d3e\": container with ID starting with f8fdc7c7e76ac57327c4821c2aba0b49d314b92c8c4ec8c2b21ea0c34c948d3e not found: ID does not exist" containerID="f8fdc7c7e76ac57327c4821c2aba0b49d314b92c8c4ec8c2b21ea0c34c948d3e" Nov 03 03:07:06.375172 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.375088 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8fdc7c7e76ac57327c4821c2aba0b49d314b92c8c4ec8c2b21ea0c34c948d3e"} err="failed to get container status \"f8fdc7c7e76ac57327c4821c2aba0b49d314b92c8c4ec8c2b21ea0c34c948d3e\": rpc error: code = NotFound desc = could not find container \"f8fdc7c7e76ac57327c4821c2aba0b49d314b92c8c4ec8c2b21ea0c34c948d3e\": container with ID starting with f8fdc7c7e76ac57327c4821c2aba0b49d314b92c8c4ec8c2b21ea0c34c948d3e not found: ID does not exist" Nov 03 03:07:06.375172 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.375107 5463 scope.go:122] "RemoveContainer" containerID="19cc3c154223e62809d13879611a241d48b394040fda141285fb99509e0c1346" Nov 03 03:07:06.387225 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.387172 5463 scope.go:122] "RemoveContainer" containerID="19cc3c154223e62809d13879611a241d48b394040fda141285fb99509e0c1346" Nov 03 03:07:06.387488 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:07:06.387470 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19cc3c154223e62809d13879611a241d48b394040fda141285fb99509e0c1346\": container with ID starting with 19cc3c154223e62809d13879611a241d48b394040fda141285fb99509e0c1346 not found: ID does not exist" containerID="19cc3c154223e62809d13879611a241d48b394040fda141285fb99509e0c1346" Nov 03 03:07:06.387569 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.387497 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19cc3c154223e62809d13879611a241d48b394040fda141285fb99509e0c1346"} err="failed to get container status \"19cc3c154223e62809d13879611a241d48b394040fda141285fb99509e0c1346\": rpc error: code = NotFound desc = could not find container \"19cc3c154223e62809d13879611a241d48b394040fda141285fb99509e0c1346\": container with ID starting with 19cc3c154223e62809d13879611a241d48b394040fda141285fb99509e0c1346 not found: ID does not exist" Nov 03 03:07:06.525231 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.525184 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw"] Nov 03 03:07:06.575335 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.575307 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-85e72378-6b4b5564bf-b8vqw"] Nov 03 03:07:06.648186 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.648149 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n"] Nov 03 03:07:06.654596 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:06.654564 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85e72378-558f5c78cf-s8h4n"] Nov 03 03:07:07.227230 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.227157 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 03:07:07.292337 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.292307 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-istio-data\") pod \"04066f9c-931a-4c65-a714-57270768f0e5\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " Nov 03 03:07:07.292537 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.292344 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-workload-socket\") pod \"04066f9c-931a-4c65-a714-57270768f0e5\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " Nov 03 03:07:07.292537 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.292378 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04066f9c-931a-4c65-a714-57270768f0e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04066f9c-931a-4c65-a714-57270768f0e5-istio-podinfo\") pod \"04066f9c-931a-4c65-a714-57270768f0e5\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " Nov 03 03:07:07.292537 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.292415 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04066f9c-931a-4c65-a714-57270768f0e5-istio-token\" (UniqueName: \"kubernetes.io/projected/04066f9c-931a-4c65-a714-57270768f0e5-istio-token\") pod \"04066f9c-931a-4c65-a714-57270768f0e5\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " Nov 03 03:07:07.292537 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.292451 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04066f9c-931a-4c65-a714-57270768f0e5-kube-api-access-7z89x\" (UniqueName: \"kubernetes.io/projected/04066f9c-931a-4c65-a714-57270768f0e5-kube-api-access-7z89x\") pod \"04066f9c-931a-4c65-a714-57270768f0e5\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " Nov 03 03:07:07.292537 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.292488 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-istio-envoy\") pod \"04066f9c-931a-4c65-a714-57270768f0e5\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " Nov 03 03:07:07.292850 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.292550 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-workload-certs\") pod \"04066f9c-931a-4c65-a714-57270768f0e5\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " Nov 03 03:07:07.292850 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.292621 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-workload-socket" pod "04066f9c-931a-4c65-a714-57270768f0e5" (UID: "04066f9c-931a-4c65-a714-57270768f0e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:07:07.292850 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.292639 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-istio-data" pod "04066f9c-931a-4c65-a714-57270768f0e5" (UID: "04066f9c-931a-4c65-a714-57270768f0e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:07:07.292850 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.292690 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-credential-socket\") pod \"04066f9c-931a-4c65-a714-57270768f0e5\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " Nov 03 03:07:07.292850 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.292740 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04066f9c-931a-4c65-a714-57270768f0e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04066f9c-931a-4c65-a714-57270768f0e5-istiod-ca-cert\") pod \"04066f9c-931a-4c65-a714-57270768f0e5\" (UID: \"04066f9c-931a-4c65-a714-57270768f0e5\") " Nov 03 03:07:07.293142 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.292929 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-workload-certs" pod "04066f9c-931a-4c65-a714-57270768f0e5" (UID: "04066f9c-931a-4c65-a714-57270768f0e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:07:07.293142 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.293108 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-credential-socket" pod "04066f9c-931a-4c65-a714-57270768f0e5" (UID: "04066f9c-931a-4c65-a714-57270768f0e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:07:07.293377 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.293352 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04066f9c-931a-4c65-a714-57270768f0e5-istiod-ca-cert" pod "04066f9c-931a-4c65-a714-57270768f0e5" (UID: "04066f9c-931a-4c65-a714-57270768f0e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:07:07.293593 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.293568 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:07.293679 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.293598 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:07.293679 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.293613 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:07.293679 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.293628 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:07.293679 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.293642 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04066f9c-931a-4c65-a714-57270768f0e5-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:07.295061 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.295039 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-istio-envoy" pod "04066f9c-931a-4c65-a714-57270768f0e5" (UID: "04066f9c-931a-4c65-a714-57270768f0e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:07:07.295262 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.295182 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04066f9c-931a-4c65-a714-57270768f0e5-istio-podinfo" pod "04066f9c-931a-4c65-a714-57270768f0e5" (UID: "04066f9c-931a-4c65-a714-57270768f0e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:07:07.295526 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.295444 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04066f9c-931a-4c65-a714-57270768f0e5-istio-token" pod "04066f9c-931a-4c65-a714-57270768f0e5" (UID: "04066f9c-931a-4c65-a714-57270768f0e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:07:07.295622 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.295531 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04066f9c-931a-4c65-a714-57270768f0e5-kube-api-access-7z89x" pod "04066f9c-931a-4c65-a714-57270768f0e5" (UID: "04066f9c-931a-4c65-a714-57270768f0e5"). InnerVolumeSpecName "kube-api-access-7z89x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:07:07.369759 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.369731 5463 generic.go:356] "Generic (PLEG): container finished" podID="04066f9c-931a-4c65-a714-57270768f0e5" containerID="dd45e4e9d6e4d5e77487e402bfedcf23814c8cdfc921a9e1c099a3dcd7e12867" exitCode=0 Nov 03 03:07:07.369880 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.369773 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" event={"ID":"04066f9c-931a-4c65-a714-57270768f0e5","Type":"ContainerDied","Data":"dd45e4e9d6e4d5e77487e402bfedcf23814c8cdfc921a9e1c099a3dcd7e12867"} Nov 03 03:07:07.369880 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.369787 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" Nov 03 03:07:07.369880 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.369853 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj" event={"ID":"04066f9c-931a-4c65-a714-57270768f0e5","Type":"ContainerDied","Data":"57e878d91634251272fe505a1a3bdcc4acb05b23b8dbdc45181ffe1fcc7dab19"} Nov 03 03:07:07.370032 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.369884 5463 scope.go:122] "RemoveContainer" containerID="dd45e4e9d6e4d5e77487e402bfedcf23814c8cdfc921a9e1c099a3dcd7e12867" Nov 03 03:07:07.382628 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.382608 5463 scope.go:122] "RemoveContainer" containerID="9188fa87fe3fa573942fbd56639310a8913d4380d4bd2f47c085360a8817826f" Nov 03 03:07:07.392343 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.392325 5463 scope.go:122] "RemoveContainer" containerID="dd45e4e9d6e4d5e77487e402bfedcf23814c8cdfc921a9e1c099a3dcd7e12867" Nov 03 03:07:07.392586 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:07:07.392564 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd45e4e9d6e4d5e77487e402bfedcf23814c8cdfc921a9e1c099a3dcd7e12867\": container with ID starting with dd45e4e9d6e4d5e77487e402bfedcf23814c8cdfc921a9e1c099a3dcd7e12867 not found: ID does not exist" containerID="dd45e4e9d6e4d5e77487e402bfedcf23814c8cdfc921a9e1c099a3dcd7e12867" Nov 03 03:07:07.392658 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.392593 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd45e4e9d6e4d5e77487e402bfedcf23814c8cdfc921a9e1c099a3dcd7e12867"} err="failed to get container status \"dd45e4e9d6e4d5e77487e402bfedcf23814c8cdfc921a9e1c099a3dcd7e12867\": rpc error: code = NotFound desc = could not find container \"dd45e4e9d6e4d5e77487e402bfedcf23814c8cdfc921a9e1c099a3dcd7e12867\": container with ID starting with dd45e4e9d6e4d5e77487e402bfedcf23814c8cdfc921a9e1c099a3dcd7e12867 not found: ID does not exist" Nov 03 03:07:07.392658 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.392613 5463 scope.go:122] "RemoveContainer" containerID="9188fa87fe3fa573942fbd56639310a8913d4380d4bd2f47c085360a8817826f" Nov 03 03:07:07.392845 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:07:07.392826 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9188fa87fe3fa573942fbd56639310a8913d4380d4bd2f47c085360a8817826f\": container with ID starting with 9188fa87fe3fa573942fbd56639310a8913d4380d4bd2f47c085360a8817826f not found: ID does not exist" containerID="9188fa87fe3fa573942fbd56639310a8913d4380d4bd2f47c085360a8817826f" Nov 03 03:07:07.392911 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.392849 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9188fa87fe3fa573942fbd56639310a8913d4380d4bd2f47c085360a8817826f"} err="failed to get container status \"9188fa87fe3fa573942fbd56639310a8913d4380d4bd2f47c085360a8817826f\": rpc error: code = NotFound desc = could not find container \"9188fa87fe3fa573942fbd56639310a8913d4380d4bd2f47c085360a8817826f\": container with ID starting with 9188fa87fe3fa573942fbd56639310a8913d4380d4bd2f47c085360a8817826f not found: ID does not exist" Nov 03 03:07:07.394414 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.394392 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04066f9c-931a-4c65-a714-57270768f0e5-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:07.394414 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.394412 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04066f9c-931a-4c65-a714-57270768f0e5-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:07.394582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.394422 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z89x\" (UniqueName: \"kubernetes.io/projected/04066f9c-931a-4c65-a714-57270768f0e5-kube-api-access-7z89x\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:07.394582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.394433 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04066f9c-931a-4c65-a714-57270768f0e5-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:07.408705 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.408684 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:07:07.421385 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.421364 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:07:07.507025 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.506949 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj"] Nov 03 03:07:07.509899 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:07.509876 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-965a07-openshift-default-5c59b9bdf-95sqj"] Nov 03 03:07:08.390117 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:08.390084 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:07:08.444063 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:08.444032 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04066f9c-931a-4c65-a714-57270768f0e5" path="/var/lib/kubelet/pods/04066f9c-931a-4c65-a714-57270768f0e5/volumes" Nov 03 03:07:08.444501 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:08.444486 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ce4448-0768-4a3c-9f54-a0a377e944bf" path="/var/lib/kubelet/pods/18ce4448-0768-4a3c-9f54-a0a377e944bf/volumes" Nov 03 03:07:08.444854 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:08.444842 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="593ad14d-f2d5-4cc8-96fb-41e6f1253b72" path="/var/lib/kubelet/pods/593ad14d-f2d5-4cc8-96fb-41e6f1253b72/volumes" Nov 03 03:07:09.015378 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:09.015287 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:09.297110 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:09.297025 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k"] Nov 03 03:07:09.404121 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:09.404091 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h"] Nov 03 03:07:09.670407 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:09.670379 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:09.829793 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:09.829749 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:11.119346 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:11.119309 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:13.010681 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:07:13.010646 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:07:13.050325 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:13.050293 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:14.297428 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:14.297391 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k" podUID="8390f42a-ae1c-4a3d-94ee-8fade238051e" containerName="payload-processing" containerID="cri-o://ec3ead13071c6d94c11614c610dfe594441c992eee877d71d48f48f291d4436d" gracePeriod=25 Nov 03 03:07:14.405343 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:14.405307 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h" podUID="428a7cac-d801-42a2-8e40-eb7a1e4c9657" containerName="payload-pre-processing" containerID="cri-o://d2846a5de06b801f540f138f981cf88859644c20bd63178c1742d4040a212439" gracePeriod=25 Nov 03 03:07:15.079335 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:15.079300 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:16.021715 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:16.021678 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:20.286128 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:20.286081 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k" podUID="8390f42a-ae1c-4a3d-94ee-8fade238051e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:07:20.286560 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:20.286130 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h" podUID="428a7cac-d801-42a2-8e40-eb7a1e4c9657" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:07:20.770159 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:20.770116 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:25.550997 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:25.550959 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:27.942509 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:27.942469 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:30.286772 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:30.286732 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h" podUID="428a7cac-d801-42a2-8e40-eb7a1e4c9657" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:07:30.287164 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:30.286731 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k" podUID="8390f42a-ae1c-4a3d-94ee-8fade238051e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:07:30.333387 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:30.333354 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:34.567748 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:34.567697 5463 generic.go:356] "Generic (PLEG): container finished" podID="8390f42a-ae1c-4a3d-94ee-8fade238051e" containerID="ec3ead13071c6d94c11614c610dfe594441c992eee877d71d48f48f291d4436d" exitCode=0 Nov 03 03:07:34.568158 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:34.568020 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k" event={"ID":"8390f42a-ae1c-4a3d-94ee-8fade238051e","Type":"ContainerDied","Data":"ec3ead13071c6d94c11614c610dfe594441c992eee877d71d48f48f291d4436d"} Nov 03 03:07:34.583399 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:34.583373 5463 generic.go:356] "Generic (PLEG): container finished" podID="428a7cac-d801-42a2-8e40-eb7a1e4c9657" containerID="d2846a5de06b801f540f138f981cf88859644c20bd63178c1742d4040a212439" exitCode=0 Nov 03 03:07:34.583522 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:34.583418 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h" event={"ID":"428a7cac-d801-42a2-8e40-eb7a1e4c9657","Type":"ContainerDied","Data":"d2846a5de06b801f540f138f981cf88859644c20bd63178c1742d4040a212439"} Nov 03 03:07:34.590757 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:34.590736 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k" Nov 03 03:07:34.629136 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:34.629107 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8390f42a-ae1c-4a3d-94ee-8fade238051e-kube-api-access-9tjmk\" (UniqueName: \"kubernetes.io/projected/8390f42a-ae1c-4a3d-94ee-8fade238051e-kube-api-access-9tjmk\") pod \"8390f42a-ae1c-4a3d-94ee-8fade238051e\" (UID: \"8390f42a-ae1c-4a3d-94ee-8fade238051e\") " Nov 03 03:07:34.629329 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:34.629251 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8390f42a-ae1c-4a3d-94ee-8fade238051e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8390f42a-ae1c-4a3d-94ee-8fade238051e-plugins-config-volume\") pod \"8390f42a-ae1c-4a3d-94ee-8fade238051e\" (UID: \"8390f42a-ae1c-4a3d-94ee-8fade238051e\") " Nov 03 03:07:34.629685 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:34.629655 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8390f42a-ae1c-4a3d-94ee-8fade238051e-plugins-config-volume" pod "8390f42a-ae1c-4a3d-94ee-8fade238051e" (UID: "8390f42a-ae1c-4a3d-94ee-8fade238051e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:07:34.631934 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:34.631906 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8390f42a-ae1c-4a3d-94ee-8fade238051e-kube-api-access-9tjmk" pod "8390f42a-ae1c-4a3d-94ee-8fade238051e" (UID: "8390f42a-ae1c-4a3d-94ee-8fade238051e"). InnerVolumeSpecName "kube-api-access-9tjmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:07:34.676212 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:34.676173 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h" Nov 03 03:07:34.730947 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:34.730895 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428a7cac-d801-42a2-8e40-eb7a1e4c9657-kube-api-access-xgmql\" (UniqueName: \"kubernetes.io/projected/428a7cac-d801-42a2-8e40-eb7a1e4c9657-kube-api-access-xgmql\") pod \"428a7cac-d801-42a2-8e40-eb7a1e4c9657\" (UID: \"428a7cac-d801-42a2-8e40-eb7a1e4c9657\") " Nov 03 03:07:34.731230 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:34.731186 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428a7cac-d801-42a2-8e40-eb7a1e4c9657-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428a7cac-d801-42a2-8e40-eb7a1e4c9657-plugins-config-volume\") pod \"428a7cac-d801-42a2-8e40-eb7a1e4c9657\" (UID: \"428a7cac-d801-42a2-8e40-eb7a1e4c9657\") " Nov 03 03:07:34.731516 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:34.731492 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428a7cac-d801-42a2-8e40-eb7a1e4c9657-plugins-config-volume" pod "428a7cac-d801-42a2-8e40-eb7a1e4c9657" (UID: "428a7cac-d801-42a2-8e40-eb7a1e4c9657"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:07:34.731720 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:34.731703 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8390f42a-ae1c-4a3d-94ee-8fade238051e-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:34.731792 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:34.731722 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428a7cac-d801-42a2-8e40-eb7a1e4c9657-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:34.731792 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:34.731731 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tjmk\" (UniqueName: \"kubernetes.io/projected/8390f42a-ae1c-4a3d-94ee-8fade238051e-kube-api-access-9tjmk\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:34.733585 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:34.733563 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428a7cac-d801-42a2-8e40-eb7a1e4c9657-kube-api-access-xgmql" pod "428a7cac-d801-42a2-8e40-eb7a1e4c9657" (UID: "428a7cac-d801-42a2-8e40-eb7a1e4c9657"). InnerVolumeSpecName "kube-api-access-xgmql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:07:34.833026 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:34.832954 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgmql\" (UniqueName: \"kubernetes.io/projected/428a7cac-d801-42a2-8e40-eb7a1e4c9657-kube-api-access-xgmql\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:35.139136 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:35.139047 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:35.591475 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:35.591447 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k" Nov 03 03:07:35.591975 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:35.591494 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k" event={"ID":"8390f42a-ae1c-4a3d-94ee-8fade238051e","Type":"ContainerDied","Data":"0ea8e841a323dc344f68f2f4f765a570ce5c5dced539e801a41528567b12010d"} Nov 03 03:07:35.591975 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:35.591563 5463 scope.go:122] "RemoveContainer" containerID="ec3ead13071c6d94c11614c610dfe594441c992eee877d71d48f48f291d4436d" Nov 03 03:07:35.593467 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:35.593425 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h" event={"ID":"428a7cac-d801-42a2-8e40-eb7a1e4c9657","Type":"ContainerDied","Data":"6abfbe5dd9b1875f6c60359f7172a26a23394c657fd7ab50966931930a0e4f43"} Nov 03 03:07:35.593467 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:35.593442 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h" Nov 03 03:07:35.602299 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:35.602279 5463 scope.go:122] "RemoveContainer" containerID="d2846a5de06b801f540f138f981cf88859644c20bd63178c1742d4040a212439" Nov 03 03:07:35.758842 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:35.758800 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k"] Nov 03 03:07:35.807256 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:35.807224 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7273af-6575df5b45-5mn2k"] Nov 03 03:07:35.918770 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:35.918741 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h"] Nov 03 03:07:35.922060 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:35.922041 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7273af-7c4d68bc5-mc44h"] Nov 03 03:07:36.443852 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:36.443822 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428a7cac-d801-42a2-8e40-eb7a1e4c9657" path="/var/lib/kubelet/pods/428a7cac-d801-42a2-8e40-eb7a1e4c9657/volumes" Nov 03 03:07:36.444281 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:36.444267 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8390f42a-ae1c-4a3d-94ee-8fade238051e" path="/var/lib/kubelet/pods/8390f42a-ae1c-4a3d-94ee-8fade238051e/volumes" Nov 03 03:07:39.974663 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:39.973893 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:43.009312 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:07:43.009277 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:07:44.192871 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.192838 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm"] Nov 03 03:07:44.194027 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.193333 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" podUID="e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c" containerName="maas-api" containerID="cri-o://e4ca537a65d442533402cd7eb1a02e06c05ddb1664bb93371ef01e8fec1899e5" gracePeriod=30 Nov 03 03:07:44.316103 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.316069 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:44.485392 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.485365 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" Nov 03 03:07:44.549558 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.549527 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-kube-api-access-894hh\" (UniqueName: \"kubernetes.io/projected/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-kube-api-access-894hh\") pod \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\" (UID: \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\") " Nov 03 03:07:44.549713 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.549604 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-openshift-service-ca\") pod \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\" (UID: \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\") " Nov 03 03:07:44.549768 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.549726 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-metrics-certs\") pod \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\" (UID: \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\") " Nov 03 03:07:44.549818 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.549779 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-maas-api-tls\") pod \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\" (UID: \"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c\") " Nov 03 03:07:44.550039 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.550008 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-openshift-service-ca" pod "e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c" (UID: "e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:07:44.550343 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.550324 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:44.552296 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.552261 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-metrics-certs" pod "e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c" (UID: "e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:07:44.552453 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.552431 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-kube-api-access-894hh" pod "e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c" (UID: "e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c"). InnerVolumeSpecName "kube-api-access-894hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:07:44.552549 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.552481 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-maas-api-tls" pod "e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c" (UID: "e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:07:44.651642 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.651605 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:44.651642 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.651643 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:44.651903 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.651660 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-894hh\" (UniqueName: \"kubernetes.io/projected/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c-kube-api-access-894hh\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:07:44.656878 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.656846 5463 generic.go:356] "Generic (PLEG): container finished" podID="e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c" containerID="e4ca537a65d442533402cd7eb1a02e06c05ddb1664bb93371ef01e8fec1899e5" exitCode=0 Nov 03 03:07:44.657038 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.656906 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" Nov 03 03:07:44.657038 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.656946 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" event={"ID":"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c","Type":"ContainerDied","Data":"e4ca537a65d442533402cd7eb1a02e06c05ddb1664bb93371ef01e8fec1899e5"} Nov 03 03:07:44.657038 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.657002 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm" event={"ID":"e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c","Type":"ContainerDied","Data":"c4f617dae9e8097498bb1b31d225f4891e637c88131529610eb9a1b2277a37fd"} Nov 03 03:07:44.657038 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.657026 5463 scope.go:122] "RemoveContainer" containerID="e4ca537a65d442533402cd7eb1a02e06c05ddb1664bb93371ef01e8fec1899e5" Nov 03 03:07:44.667566 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.667547 5463 scope.go:122] "RemoveContainer" containerID="e4ca537a65d442533402cd7eb1a02e06c05ddb1664bb93371ef01e8fec1899e5" Nov 03 03:07:44.667923 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:07:44.667821 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4ca537a65d442533402cd7eb1a02e06c05ddb1664bb93371ef01e8fec1899e5\": container with ID starting with e4ca537a65d442533402cd7eb1a02e06c05ddb1664bb93371ef01e8fec1899e5 not found: ID does not exist" containerID="e4ca537a65d442533402cd7eb1a02e06c05ddb1664bb93371ef01e8fec1899e5" Nov 03 03:07:44.667923 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.667848 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ca537a65d442533402cd7eb1a02e06c05ddb1664bb93371ef01e8fec1899e5"} err="failed to get container status \"e4ca537a65d442533402cd7eb1a02e06c05ddb1664bb93371ef01e8fec1899e5\": rpc error: code = NotFound desc = could not find container \"e4ca537a65d442533402cd7eb1a02e06c05ddb1664bb93371ef01e8fec1899e5\": container with ID starting with e4ca537a65d442533402cd7eb1a02e06c05ddb1664bb93371ef01e8fec1899e5 not found: ID does not exist" Nov 03 03:07:44.822818 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.822748 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm"] Nov 03 03:07:44.825894 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:44.825871 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85e72378-6c557d985d-7v8pm"] Nov 03 03:07:46.034869 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:46.034832 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:46.444058 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:46.444023 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c" path="/var/lib/kubelet/pods/e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c/volumes" Nov 03 03:07:50.863624 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:50.863546 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:53.918696 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:53.918659 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:54.084492 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:54.084457 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:55.550926 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:55.550883 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:55.815223 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:55.815110 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:56.194008 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:56.193975 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:56.690744 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:07:56.690711 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:00.320580 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:00.320544 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:01.641841 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:01.641806 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q"] Nov 03 03:08:01.652585 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:01.652563 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2"] Nov 03 03:08:01.816616 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:01.816585 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:02.071852 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:02.071820 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:04.946452 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:04.946415 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:05.200781 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:05.200698 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:06.697764 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:06.697725 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:07.145682 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:07.145651 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:07.174259 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:07.174232 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54"] Nov 03 03:08:07.174480 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:07.174460 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" podUID="b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" containerName="istio-proxy" containerID="cri-o://b73a5b35a90e2f33a5ad66801f7ffc9a1ec78093c2b50c18c6855e40048d936a" gracePeriod=30 Nov 03 03:08:07.322120 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:07.322086 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:07.667808 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:07.667776 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:08.042056 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:08.041978 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:12.071219 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.071170 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:12.468755 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.468721 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 03:08:12.587178 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.587137 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-workload-socket\") pod \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " Nov 03 03:08:12.587394 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.587209 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istiod-ca-cert\") pod \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " Nov 03 03:08:12.587394 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.587246 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-envoy\") pod \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " Nov 03 03:08:12.587394 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.587296 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-data\") pod \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " Nov 03 03:08:12.587394 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.587344 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-kube-api-access-d7tpf\" (UniqueName: \"kubernetes.io/projected/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-kube-api-access-d7tpf\") pod \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " Nov 03 03:08:12.587394 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.587385 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-token\" (UniqueName: \"kubernetes.io/projected/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-token\") pod \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " Nov 03 03:08:12.587701 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.587431 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-podinfo\") pod \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " Nov 03 03:08:12.587701 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.587445 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-workload-socket" pod "b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" (UID: "b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:12.587701 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.587485 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-workload-certs\") pod \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " Nov 03 03:08:12.587701 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.587534 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-credential-socket\") pod \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\" (UID: \"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a\") " Nov 03 03:08:12.587927 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.587787 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-data" pod "b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" (UID: "b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:12.587927 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.587819 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istiod-ca-cert" pod "b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" (UID: "b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:08:12.588056 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.588031 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-credential-socket" pod "b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" (UID: "b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:12.588114 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.587945 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-workload-certs" pod "b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" (UID: "b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:12.588258 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.588223 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:12.588258 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.588241 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:12.588258 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.588254 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:12.588456 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.588268 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:12.590006 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.589968 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-token" pod "b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" (UID: "b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:12.590241 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.590217 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-kube-api-access-d7tpf" pod "b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" (UID: "b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a"). InnerVolumeSpecName "kube-api-access-d7tpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:12.590413 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.590386 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-envoy" pod "b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" (UID: "b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:12.590514 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.590391 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-podinfo" pod "b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" (UID: "b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:08:12.689772 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.689733 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7tpf\" (UniqueName: \"kubernetes.io/projected/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-kube-api-access-d7tpf\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:12.689772 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.689767 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:12.690007 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.689781 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:12.690007 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.689794 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:12.690007 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.689806 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:12.764100 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.764061 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:12.853712 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.853673 5463 generic.go:356] "Generic (PLEG): container finished" podID="b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" containerID="b73a5b35a90e2f33a5ad66801f7ffc9a1ec78093c2b50c18c6855e40048d936a" exitCode=0 Nov 03 03:08:12.853907 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.853731 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" Nov 03 03:08:12.853907 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.853754 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" event={"ID":"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a","Type":"ContainerDied","Data":"b73a5b35a90e2f33a5ad66801f7ffc9a1ec78093c2b50c18c6855e40048d936a"} Nov 03 03:08:12.853907 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.853812 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54" event={"ID":"b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a","Type":"ContainerDied","Data":"cdddfa897e7ad3fd41b4c7eb9be8ecd3d2fe4b6c014a3f47b4c47e268a0c6e69"} Nov 03 03:08:12.853907 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.853841 5463 scope.go:122] "RemoveContainer" containerID="b73a5b35a90e2f33a5ad66801f7ffc9a1ec78093c2b50c18c6855e40048d936a" Nov 03 03:08:12.867498 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.867477 5463 scope.go:122] "RemoveContainer" containerID="7f7d2be25db222b2001948ef40d630d02d2bd4dfb47b2693fad9596345175625" Nov 03 03:08:12.877760 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.877741 5463 scope.go:122] "RemoveContainer" containerID="b73a5b35a90e2f33a5ad66801f7ffc9a1ec78093c2b50c18c6855e40048d936a" Nov 03 03:08:12.878043 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:08:12.878021 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b73a5b35a90e2f33a5ad66801f7ffc9a1ec78093c2b50c18c6855e40048d936a\": container with ID starting with b73a5b35a90e2f33a5ad66801f7ffc9a1ec78093c2b50c18c6855e40048d936a not found: ID does not exist" containerID="b73a5b35a90e2f33a5ad66801f7ffc9a1ec78093c2b50c18c6855e40048d936a" Nov 03 03:08:12.878116 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.878050 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73a5b35a90e2f33a5ad66801f7ffc9a1ec78093c2b50c18c6855e40048d936a"} err="failed to get container status \"b73a5b35a90e2f33a5ad66801f7ffc9a1ec78093c2b50c18c6855e40048d936a\": rpc error: code = NotFound desc = could not find container \"b73a5b35a90e2f33a5ad66801f7ffc9a1ec78093c2b50c18c6855e40048d936a\": container with ID starting with b73a5b35a90e2f33a5ad66801f7ffc9a1ec78093c2b50c18c6855e40048d936a not found: ID does not exist" Nov 03 03:08:12.878116 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.878069 5463 scope.go:122] "RemoveContainer" containerID="7f7d2be25db222b2001948ef40d630d02d2bd4dfb47b2693fad9596345175625" Nov 03 03:08:12.878377 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:08:12.878349 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f7d2be25db222b2001948ef40d630d02d2bd4dfb47b2693fad9596345175625\": container with ID starting with 7f7d2be25db222b2001948ef40d630d02d2bd4dfb47b2693fad9596345175625 not found: ID does not exist" containerID="7f7d2be25db222b2001948ef40d630d02d2bd4dfb47b2693fad9596345175625" Nov 03 03:08:12.878445 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:12.878373 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f7d2be25db222b2001948ef40d630d02d2bd4dfb47b2693fad9596345175625"} err="failed to get container status \"7f7d2be25db222b2001948ef40d630d02d2bd4dfb47b2693fad9596345175625\": rpc error: code = NotFound desc = could not find container \"7f7d2be25db222b2001948ef40d630d02d2bd4dfb47b2693fad9596345175625\": container with ID starting with 7f7d2be25db222b2001948ef40d630d02d2bd4dfb47b2693fad9596345175625 not found: ID does not exist" Nov 03 03:08:13.006998 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:08:13.006910 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:08:13.021361 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:13.021335 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54"] Nov 03 03:08:13.026724 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:13.026704 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7273af-openshift-default-74d84dfd9f-n2c54"] Nov 03 03:08:14.443605 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:14.443571 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" path="/var/lib/kubelet/pods/b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a/volumes" Nov 03 03:08:14.594658 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:14.594621 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:15.372418 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:15.372386 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7"] Nov 03 03:08:15.649832 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:15.649798 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:15.803697 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:15.803662 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:16.678559 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:16.678514 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" containerID="cri-o://cabe768d8d0d50ac38c42d8e4a8c830de5671354ea82a7ca547f11f1601e6c4e" gracePeriod=45 Nov 03 03:08:16.688460 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:16.688425 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" containerID="cri-o://d48027a489669f17801f1913a3155d0898ef687f1cccfa94f109b7e6177de1f0" gracePeriod=45 Nov 03 03:08:16.818336 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:16.818300 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": dial tcp 10.133.0.71:8000: connect: connection refused" Nov 03 03:08:16.900481 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:16.899468 5463 generic.go:356] "Generic (PLEG): container finished" podID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerID="d48027a489669f17801f1913a3155d0898ef687f1cccfa94f109b7e6177de1f0" exitCode=0 Nov 03 03:08:16.900481 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:16.899565 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" event={"ID":"e2f08835-5113-4ca7-9cfa-e107685e3348","Type":"ContainerDied","Data":"d48027a489669f17801f1913a3155d0898ef687f1cccfa94f109b7e6177de1f0"} Nov 03 03:08:16.915954 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:16.907828 5463 generic.go:356] "Generic (PLEG): container finished" podID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerID="cabe768d8d0d50ac38c42d8e4a8c830de5671354ea82a7ca547f11f1601e6c4e" exitCode=0 Nov 03 03:08:16.915954 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:16.907943 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" event={"ID":"3e2af92a-6565-45d7-ae87-ef5809aa8517","Type":"ContainerDied","Data":"cabe768d8d0d50ac38c42d8e4a8c830de5671354ea82a7ca547f11f1601e6c4e"} Nov 03 03:08:17.049576 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.049543 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 03:08:17.085878 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.085848 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 03:08:17.143626 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.143594 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-home\" (UniqueName: \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-home\") pod \"e2f08835-5113-4ca7-9cfa-e107685e3348\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " Nov 03 03:08:17.143899 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.143674 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f08835-5113-4ca7-9cfa-e107685e3348-kube-api-access-98szb\" (UniqueName: \"kubernetes.io/projected/e2f08835-5113-4ca7-9cfa-e107685e3348-kube-api-access-98szb\") pod \"e2f08835-5113-4ca7-9cfa-e107685e3348\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " Nov 03 03:08:17.143899 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.143780 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-model-cache\") pod \"3e2af92a-6565-45d7-ae87-ef5809aa8517\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " Nov 03 03:08:17.143899 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.143808 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2f08835-5113-4ca7-9cfa-e107685e3348-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2f08835-5113-4ca7-9cfa-e107685e3348-tls-certs\") pod \"e2f08835-5113-4ca7-9cfa-e107685e3348\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " Nov 03 03:08:17.143899 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.143855 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-model-cache\") pod \"e2f08835-5113-4ca7-9cfa-e107685e3348\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " Nov 03 03:08:17.143899 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.143887 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-dshm\") pod \"3e2af92a-6565-45d7-ae87-ef5809aa8517\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " Nov 03 03:08:17.144293 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.143904 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-home" pod "e2f08835-5113-4ca7-9cfa-e107685e3348" (UID: "e2f08835-5113-4ca7-9cfa-e107685e3348"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:17.144293 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.143924 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-home\" (UniqueName: \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-home\") pod \"3e2af92a-6565-45d7-ae87-ef5809aa8517\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " Nov 03 03:08:17.144293 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.143976 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-dshm\") pod \"e2f08835-5113-4ca7-9cfa-e107685e3348\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " Nov 03 03:08:17.144293 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.144048 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e2af92a-6565-45d7-ae87-ef5809aa8517-tls-certs\" (UniqueName: \"kubernetes.io/secret/3e2af92a-6565-45d7-ae87-ef5809aa8517-tls-certs\") pod \"3e2af92a-6565-45d7-ae87-ef5809aa8517\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " Nov 03 03:08:17.144293 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.144088 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e2af92a-6565-45d7-ae87-ef5809aa8517-kube-api-access-gcc6q\" (UniqueName: \"kubernetes.io/projected/3e2af92a-6565-45d7-ae87-ef5809aa8517-kube-api-access-gcc6q\") pod \"3e2af92a-6565-45d7-ae87-ef5809aa8517\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " Nov 03 03:08:17.144293 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.144125 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-tmp-dir\") pod \"3e2af92a-6565-45d7-ae87-ef5809aa8517\" (UID: \"3e2af92a-6565-45d7-ae87-ef5809aa8517\") " Nov 03 03:08:17.144293 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.144146 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-model-cache" pod "3e2af92a-6565-45d7-ae87-ef5809aa8517" (UID: "3e2af92a-6565-45d7-ae87-ef5809aa8517"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:17.144293 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.144156 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-tmp-dir\") pod \"e2f08835-5113-4ca7-9cfa-e107685e3348\" (UID: \"e2f08835-5113-4ca7-9cfa-e107685e3348\") " Nov 03 03:08:17.144293 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.144171 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-model-cache" pod "e2f08835-5113-4ca7-9cfa-e107685e3348" (UID: "e2f08835-5113-4ca7-9cfa-e107685e3348"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:17.145420 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.144661 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-tmp-dir" pod "3e2af92a-6565-45d7-ae87-ef5809aa8517" (UID: "3e2af92a-6565-45d7-ae87-ef5809aa8517"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:17.145420 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.144799 5463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-model-cache\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.145420 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.144819 5463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-model-cache\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.145420 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.144831 5463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-tmp-dir\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.145420 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.144845 5463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-home\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.145420 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.144814 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-home" pod "3e2af92a-6565-45d7-ae87-ef5809aa8517" (UID: "3e2af92a-6565-45d7-ae87-ef5809aa8517"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:17.145420 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.144902 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-tmp-dir" pod "e2f08835-5113-4ca7-9cfa-e107685e3348" (UID: "e2f08835-5113-4ca7-9cfa-e107685e3348"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:17.146540 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.146515 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f08835-5113-4ca7-9cfa-e107685e3348-kube-api-access-98szb" pod "e2f08835-5113-4ca7-9cfa-e107685e3348" (UID: "e2f08835-5113-4ca7-9cfa-e107685e3348"). InnerVolumeSpecName "kube-api-access-98szb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:17.147089 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.147024 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-dshm" pod "3e2af92a-6565-45d7-ae87-ef5809aa8517" (UID: "3e2af92a-6565-45d7-ae87-ef5809aa8517"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:17.147242 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.147124 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2f08835-5113-4ca7-9cfa-e107685e3348-tls-certs" pod "e2f08835-5113-4ca7-9cfa-e107685e3348" (UID: "e2f08835-5113-4ca7-9cfa-e107685e3348"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:08:17.147242 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.147131 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-dshm" pod "e2f08835-5113-4ca7-9cfa-e107685e3348" (UID: "e2f08835-5113-4ca7-9cfa-e107685e3348"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:17.147951 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.147924 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e2af92a-6565-45d7-ae87-ef5809aa8517-tls-certs" pod "3e2af92a-6565-45d7-ae87-ef5809aa8517" (UID: "3e2af92a-6565-45d7-ae87-ef5809aa8517"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:08:17.148038 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.147972 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e2af92a-6565-45d7-ae87-ef5809aa8517-kube-api-access-gcc6q" pod "3e2af92a-6565-45d7-ae87-ef5809aa8517" (UID: "3e2af92a-6565-45d7-ae87-ef5809aa8517"). InnerVolumeSpecName "kube-api-access-gcc6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:17.245598 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.245526 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98szb\" (UniqueName: \"kubernetes.io/projected/e2f08835-5113-4ca7-9cfa-e107685e3348-kube-api-access-98szb\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.245598 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.245554 5463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2f08835-5113-4ca7-9cfa-e107685e3348-tls-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.245598 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.245566 5463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-dshm\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.245598 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.245576 5463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e2af92a-6565-45d7-ae87-ef5809aa8517-home\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.245598 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.245584 5463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-dshm\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.245598 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.245593 5463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e2af92a-6565-45d7-ae87-ef5809aa8517-tls-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.245598 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.245603 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcc6q\" (UniqueName: \"kubernetes.io/projected/3e2af92a-6565-45d7-ae87-ef5809aa8517-kube-api-access-gcc6q\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.245968 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.245612 5463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2f08835-5113-4ca7-9cfa-e107685e3348-tmp-dir\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:17.259360 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.259335 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:17.915176 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.915140 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" Nov 03 03:08:17.915669 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.915144 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q" event={"ID":"e2f08835-5113-4ca7-9cfa-e107685e3348","Type":"ContainerDied","Data":"4fd161cb13017f7ddc41eb65b542d0b4ba5df370cff9051405d6939db971fe9e"} Nov 03 03:08:17.915669 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.915257 5463 scope.go:122] "RemoveContainer" containerID="d48027a489669f17801f1913a3155d0898ef687f1cccfa94f109b7e6177de1f0" Nov 03 03:08:17.916626 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.916603 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" Nov 03 03:08:17.916711 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.916646 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" event={"ID":"3e2af92a-6565-45d7-ae87-ef5809aa8517","Type":"ContainerDied","Data":"957f1ccc33afcb51a8754961cacac1fbfdba59cae0c3d5ade6d2d137846c73e1"} Nov 03 03:08:17.927476 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:17.927393 5463 scope.go:122] "RemoveContainer" containerID="cabe768d8d0d50ac38c42d8e4a8c830de5671354ea82a7ca547f11f1601e6c4e" Nov 03 03:08:18.029878 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:18.029791 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:08:18.082494 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:18.082452 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q"] Nov 03 03:08:18.132781 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:18.132748 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/premium-w4-c7d758-kserve-748696cd88-d442q"] Nov 03 03:08:18.197671 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:18.197642 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2"] Nov 03 03:08:18.200147 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:18.200127 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/simulator-w4-c7d758-kserve-85f59d8fc-8jll2"] Nov 03 03:08:18.443402 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:18.443367 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" path="/var/lib/kubelet/pods/3e2af92a-6565-45d7-ae87-ef5809aa8517/volumes" Nov 03 03:08:18.443834 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:18.443820 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" path="/var/lib/kubelet/pods/e2f08835-5113-4ca7-9cfa-e107685e3348/volumes" Nov 03 03:08:18.909491 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:18.909460 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:20.372892 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.372830 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7" podUID="32df4d57-7e5b-4136-9e75-5c4e9290fd30" containerName="payload-pre-processing" containerID="cri-o://09860774bae4e8bca0e5fed3180b95a45c4fea28047e0801612cd93c1ff1e7e9" gracePeriod=25 Nov 03 03:08:20.387445 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.387414 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:20.966729 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.966700 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv"] Nov 03 03:08:20.967320 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967306 5463 cpu_manager.go:414] "containerMap: removing container" podUID="04066f9c-931a-4c65-a714-57270768f0e5" containerName="istio-proxy" Nov 03 03:08:20.967386 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967322 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04066f9c-931a-4c65-a714-57270768f0e5" containerName="istio-proxy" Nov 03 03:08:20.967386 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967342 5463 cpu_manager.go:414] "containerMap: removing container" podUID="04066f9c-931a-4c65-a714-57270768f0e5" containerName="istio-proxy" Nov 03 03:08:20.967386 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967348 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04066f9c-931a-4c65-a714-57270768f0e5" containerName="istio-proxy" Nov 03 03:08:20.967386 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967360 5463 cpu_manager.go:414] "containerMap: removing container" podUID="18ce4448-0768-4a3c-9f54-a0a377e944bf" containerName="payload-pre-processing" Nov 03 03:08:20.967386 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967365 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ce4448-0768-4a3c-9f54-a0a377e944bf" containerName="payload-pre-processing" Nov 03 03:08:20.967386 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967383 5463 cpu_manager.go:414] "containerMap: removing container" podUID="e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c" containerName="maas-api" Nov 03 03:08:20.967582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967389 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c" containerName="maas-api" Nov 03 03:08:20.967582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967396 5463 cpu_manager.go:414] "containerMap: removing container" podUID="428a7cac-d801-42a2-8e40-eb7a1e4c9657" containerName="payload-pre-processing" Nov 03 03:08:20.967582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967401 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428a7cac-d801-42a2-8e40-eb7a1e4c9657" containerName="payload-pre-processing" Nov 03 03:08:20.967582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967417 5463 cpu_manager.go:414] "containerMap: removing container" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" Nov 03 03:08:20.967582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967423 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" Nov 03 03:08:20.967582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967429 5463 cpu_manager.go:414] "containerMap: removing container" podUID="b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" containerName="istio-proxy" Nov 03 03:08:20.967582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967433 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" containerName="istio-proxy" Nov 03 03:08:20.967582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967449 5463 cpu_manager.go:414] "containerMap: removing container" podUID="8390f42a-ae1c-4a3d-94ee-8fade238051e" containerName="payload-processing" Nov 03 03:08:20.967582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967453 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8390f42a-ae1c-4a3d-94ee-8fade238051e" containerName="payload-processing" Nov 03 03:08:20.967582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967458 5463 cpu_manager.go:414] "containerMap: removing container" podUID="b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" containerName="istio-proxy" Nov 03 03:08:20.967582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967463 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" containerName="istio-proxy" Nov 03 03:08:20.967582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967469 5463 cpu_manager.go:414] "containerMap: removing container" podUID="593ad14d-f2d5-4cc8-96fb-41e6f1253b72" containerName="payload-processing" Nov 03 03:08:20.967582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967473 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="593ad14d-f2d5-4cc8-96fb-41e6f1253b72" containerName="payload-processing" Nov 03 03:08:20.967582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967489 5463 cpu_manager.go:414] "containerMap: removing container" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" Nov 03 03:08:20.967582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967494 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" Nov 03 03:08:20.968039 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967588 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e2af92a-6565-45d7-ae87-ef5809aa8517" containerName="main" Nov 03 03:08:20.968039 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967598 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="04066f9c-931a-4c65-a714-57270768f0e5" containerName="istio-proxy" Nov 03 03:08:20.968039 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967605 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="04066f9c-931a-4c65-a714-57270768f0e5" containerName="istio-proxy" Nov 03 03:08:20.968039 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967612 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ce4448-0768-4a3c-9f54-a0a377e944bf" containerName="payload-pre-processing" Nov 03 03:08:20.968039 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967621 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c681fe-6b83-4e9e-a3ae-29b032bb7c1c" containerName="maas-api" Nov 03 03:08:20.968039 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967627 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="428a7cac-d801-42a2-8e40-eb7a1e4c9657" containerName="payload-pre-processing" Nov 03 03:08:20.968039 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967636 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f08835-5113-4ca7-9cfa-e107685e3348" containerName="main" Nov 03 03:08:20.968039 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967643 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" containerName="istio-proxy" Nov 03 03:08:20.968039 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967650 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="8390f42a-ae1c-4a3d-94ee-8fade238051e" containerName="payload-processing" Nov 03 03:08:20.968039 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967656 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b9bf51-7d4e-4a3d-9c86-d19e80745d2a" containerName="istio-proxy" Nov 03 03:08:20.968039 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.967662 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="593ad14d-f2d5-4cc8-96fb-41e6f1253b72" containerName="payload-processing" Nov 03 03:08:20.972347 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:20.972328 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:21.023486 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.023448 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv"] Nov 03 03:08:21.070898 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.070864 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-f735f4\"/\"kube-root-ca.crt\"" Nov 03 03:08:21.071078 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.071033 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-f735f4\"/\"sub-test-model-f735f4-kserve-self-signed-certs\"" Nov 03 03:08:21.071270 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.071092 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-f735f4\"/\"openshift-service-ca.crt\"" Nov 03 03:08:21.078126 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.078106 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-f735f4\"/\"default-dockercfg-n7nmc\"" Nov 03 03:08:21.088028 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.087997 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:21.090237 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.090184 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-tmp-dir\") pod \"sub-test-model-f735f4-kserve-85f9f7b767-5xkgv\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:21.090307 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.090277 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-home\") pod \"sub-test-model-f735f4-kserve-85f9f7b767-5xkgv\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:21.090352 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.090323 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53646ed7-2ee1-4a28-9242-033c5be233e4-tls-certs\") pod \"sub-test-model-f735f4-kserve-85f9f7b767-5xkgv\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:21.090420 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.090402 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d78fg\" (UniqueName: \"kubernetes.io/projected/53646ed7-2ee1-4a28-9242-033c5be233e4-kube-api-access-d78fg\") pod \"sub-test-model-f735f4-kserve-85f9f7b767-5xkgv\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:21.090463 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.090438 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-model-cache\") pod \"sub-test-model-f735f4-kserve-85f9f7b767-5xkgv\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:21.090503 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.090478 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-dshm\") pod \"sub-test-model-f735f4-kserve-85f9f7b767-5xkgv\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:21.191800 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.191764 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53646ed7-2ee1-4a28-9242-033c5be233e4-tls-certs\") pod \"sub-test-model-f735f4-kserve-85f9f7b767-5xkgv\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:21.192025 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.191852 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d78fg\" (UniqueName: \"kubernetes.io/projected/53646ed7-2ee1-4a28-9242-033c5be233e4-kube-api-access-d78fg\") pod \"sub-test-model-f735f4-kserve-85f9f7b767-5xkgv\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:21.192025 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.191880 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-model-cache\") pod \"sub-test-model-f735f4-kserve-85f9f7b767-5xkgv\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:21.192025 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.191945 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-dshm\") pod \"sub-test-model-f735f4-kserve-85f9f7b767-5xkgv\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:21.192258 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.192036 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-tmp-dir\") pod \"sub-test-model-f735f4-kserve-85f9f7b767-5xkgv\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:21.192258 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.192103 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-home\") pod \"sub-test-model-f735f4-kserve-85f9f7b767-5xkgv\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:21.192388 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.192347 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-model-cache\") pod \"sub-test-model-f735f4-kserve-85f9f7b767-5xkgv\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:21.192494 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.192474 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-tmp-dir\") pod \"sub-test-model-f735f4-kserve-85f9f7b767-5xkgv\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:21.192563 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.192503 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-home\") pod \"sub-test-model-f735f4-kserve-85f9f7b767-5xkgv\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:21.194222 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.194169 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-dshm\") pod \"sub-test-model-f735f4-kserve-85f9f7b767-5xkgv\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:21.194355 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.194339 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53646ed7-2ee1-4a28-9242-033c5be233e4-tls-certs\") pod \"sub-test-model-f735f4-kserve-85f9f7b767-5xkgv\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:21.249866 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.249805 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d78fg\" (UniqueName: \"kubernetes.io/projected/53646ed7-2ee1-4a28-9242-033c5be233e4-kube-api-access-d78fg\") pod \"sub-test-model-f735f4-kserve-85f9f7b767-5xkgv\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:21.284957 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.284931 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:21.659749 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:08:21.659708 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53646ed7_2ee1_4a28_9242_033c5be233e4.slice/crio-ed37cbe04148c19a97c6ad969d215b0841cb6913255b0a5c46cd537265df4302 WatchSource:0}: Error finding container ed37cbe04148c19a97c6ad969d215b0841cb6913255b0a5c46cd537265df4302: Status 404 returned error can't find the container with id ed37cbe04148c19a97c6ad969d215b0841cb6913255b0a5c46cd537265df4302 Nov 03 03:08:21.661834 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.661810 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv"] Nov 03 03:08:21.957624 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.957537 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" event={"ID":"53646ed7-2ee1-4a28-9242-033c5be233e4","Type":"ContainerStarted","Data":"1fc63374ac54e365dc96a2f0448097632f22dc9a6be8ca640e70041e31ea0412"} Nov 03 03:08:21.957624 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:21.957581 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" event={"ID":"53646ed7-2ee1-4a28-9242-033c5be233e4","Type":"ContainerStarted","Data":"ed37cbe04148c19a97c6ad969d215b0841cb6913255b0a5c46cd537265df4302"} Nov 03 03:08:22.073724 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:22.073671 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" podStartSLOduration=2.073656435 podStartE2EDuration="2.073656435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:08:22.072736405 +0000 UTC m=+4583.756845589" watchObservedRunningTime="2026-11-03 03:08:22.073656435 +0000 UTC m=+4583.757765589" Nov 03 03:08:22.235542 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:22.235470 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:22.285773 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:22.285737 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:24.432948 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:24.432912 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7" podUID="32df4d57-7e5b-4136-9e75-5c4e9290fd30" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:08:25.645974 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:25.645927 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88"] Nov 03 03:08:25.659544 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:25.659510 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:26.016638 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:26.016550 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:26.228569 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:26.228538 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:27.427822 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:27.427788 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:27.679007 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:27.678924 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:28.234821 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:28.234793 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:28.452126 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:28.452090 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:28.707140 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:28.707103 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb"] Nov 03 03:08:28.707342 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:28.707313 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" podUID="88e27111-aadb-40b3-a564-cf0b3a6f86bb" containerName="istio-proxy" containerID="cri-o://3f41768e1a9ece49aea430c3109e6ec3ac881da64df844e2b5a608dd2bace20c" gracePeriod=30 Nov 03 03:08:30.647178 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:30.647135 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88" podUID="9b7b703f-6c31-4881-8e71-a23b27c3e6e7" containerName="payload-pre-processing" containerID="cri-o://cdef056da06131eb9232cb318b2df99b2afce31099086cb00998882adae253c5" gracePeriod=25 Nov 03 03:08:31.171617 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:31.171583 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:31.285053 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:31.285015 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:31.298285 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:31.298211 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:31.368921 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:31.368891 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:31.465913 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:31.465879 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-97lfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:08:31.466084 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:31.465933 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" podUID="88e27111-aadb-40b3-a564-cf0b3a6f86bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:08:32.067655 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:32.067625 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:08:32.249979 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:32.249939 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88" podUID="9b7b703f-6c31-4881-8e71-a23b27c3e6e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:08:32.674697 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:32.674661 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:33.971027 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:33.971005 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 03:08:34.072977 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.072901 5463 generic.go:356] "Generic (PLEG): container finished" podID="88e27111-aadb-40b3-a564-cf0b3a6f86bb" containerID="3f41768e1a9ece49aea430c3109e6ec3ac881da64df844e2b5a608dd2bace20c" exitCode=0 Nov 03 03:08:34.072977 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.072945 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" Nov 03 03:08:34.072977 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.072960 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" event={"ID":"88e27111-aadb-40b3-a564-cf0b3a6f86bb","Type":"ContainerDied","Data":"3f41768e1a9ece49aea430c3109e6ec3ac881da64df844e2b5a608dd2bace20c"} Nov 03 03:08:34.073228 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.073012 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb" event={"ID":"88e27111-aadb-40b3-a564-cf0b3a6f86bb","Type":"ContainerDied","Data":"b648081012724fad1ab63b9b9bdf1630a1090ffd787cd887b32f6774c1f15d10"} Nov 03 03:08:34.073228 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.073043 5463 scope.go:122] "RemoveContainer" containerID="3f41768e1a9ece49aea430c3109e6ec3ac881da64df844e2b5a608dd2bace20c" Nov 03 03:08:34.082843 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.082817 5463 scope.go:122] "RemoveContainer" containerID="3f41768e1a9ece49aea430c3109e6ec3ac881da64df844e2b5a608dd2bace20c" Nov 03 03:08:34.083067 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:08:34.083052 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f41768e1a9ece49aea430c3109e6ec3ac881da64df844e2b5a608dd2bace20c\": container with ID starting with 3f41768e1a9ece49aea430c3109e6ec3ac881da64df844e2b5a608dd2bace20c not found: ID does not exist" containerID="3f41768e1a9ece49aea430c3109e6ec3ac881da64df844e2b5a608dd2bace20c" Nov 03 03:08:34.083116 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.083073 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f41768e1a9ece49aea430c3109e6ec3ac881da64df844e2b5a608dd2bace20c"} err="failed to get container status \"3f41768e1a9ece49aea430c3109e6ec3ac881da64df844e2b5a608dd2bace20c\": rpc error: code = NotFound desc = could not find container \"3f41768e1a9ece49aea430c3109e6ec3ac881da64df844e2b5a608dd2bace20c\": container with ID starting with 3f41768e1a9ece49aea430c3109e6ec3ac881da64df844e2b5a608dd2bace20c not found: ID does not exist" Nov 03 03:08:34.157574 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.157551 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-data\") pod \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " Nov 03 03:08:34.157741 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.157699 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e27111-aadb-40b3-a564-cf0b3a6f86bb-kube-api-access-hv4fr\" (UniqueName: \"kubernetes.io/projected/88e27111-aadb-40b3-a564-cf0b3a6f86bb-kube-api-access-hv4fr\") pod \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " Nov 03 03:08:34.157741 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.157734 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-podinfo\") pod \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " Nov 03 03:08:34.157864 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.157776 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-workload-certs\") pod \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " Nov 03 03:08:34.157864 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.157812 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-workload-socket\") pod \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " Nov 03 03:08:34.157984 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.157852 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-data" pod "88e27111-aadb-40b3-a564-cf0b3a6f86bb" (UID: "88e27111-aadb-40b3-a564-cf0b3a6f86bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.157984 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.157891 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-token\" (UniqueName: \"kubernetes.io/projected/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-token\") pod \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " Nov 03 03:08:34.157984 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.157940 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-envoy\") pod \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " Nov 03 03:08:34.158134 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.158003 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istiod-ca-cert\") pod \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " Nov 03 03:08:34.158134 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.158088 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-credential-socket\") pod \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\" (UID: \"88e27111-aadb-40b3-a564-cf0b3a6f86bb\") " Nov 03 03:08:34.158526 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.158503 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.158735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.158713 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-credential-socket" pod "88e27111-aadb-40b3-a564-cf0b3a6f86bb" (UID: "88e27111-aadb-40b3-a564-cf0b3a6f86bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.158893 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.158876 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-workload-certs" pod "88e27111-aadb-40b3-a564-cf0b3a6f86bb" (UID: "88e27111-aadb-40b3-a564-cf0b3a6f86bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.159032 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.159016 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-workload-socket" pod "88e27111-aadb-40b3-a564-cf0b3a6f86bb" (UID: "88e27111-aadb-40b3-a564-cf0b3a6f86bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.159613 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.159590 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istiod-ca-cert" pod "88e27111-aadb-40b3-a564-cf0b3a6f86bb" (UID: "88e27111-aadb-40b3-a564-cf0b3a6f86bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:08:34.161630 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.161594 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e27111-aadb-40b3-a564-cf0b3a6f86bb-kube-api-access-hv4fr" pod "88e27111-aadb-40b3-a564-cf0b3a6f86bb" (UID: "88e27111-aadb-40b3-a564-cf0b3a6f86bb"). InnerVolumeSpecName "kube-api-access-hv4fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:34.161768 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.161678 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-podinfo" pod "88e27111-aadb-40b3-a564-cf0b3a6f86bb" (UID: "88e27111-aadb-40b3-a564-cf0b3a6f86bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:08:34.162241 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.162177 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-envoy" pod "88e27111-aadb-40b3-a564-cf0b3a6f86bb" (UID: "88e27111-aadb-40b3-a564-cf0b3a6f86bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:34.162540 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.162513 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-token" pod "88e27111-aadb-40b3-a564-cf0b3a6f86bb" (UID: "88e27111-aadb-40b3-a564-cf0b3a6f86bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:34.259931 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.259886 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv4fr\" (UniqueName: \"kubernetes.io/projected/88e27111-aadb-40b3-a564-cf0b3a6f86bb-kube-api-access-hv4fr\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.260121 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.259934 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.260121 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.259962 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.260121 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.259977 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.260121 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.259992 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.260121 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.260007 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.260121 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.260022 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e27111-aadb-40b3-a564-cf0b3a6f86bb-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.260121 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.260035 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e27111-aadb-40b3-a564-cf0b3a6f86bb-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:34.432325 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.432293 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7" podUID="32df4d57-7e5b-4136-9e75-5c4e9290fd30" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:08:34.500628 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.500599 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb"] Nov 03 03:08:34.504442 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:34.504410 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-97lfb"] Nov 03 03:08:36.442805 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:36.442765 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e27111-aadb-40b3-a564-cf0b3a6f86bb" path="/var/lib/kubelet/pods/88e27111-aadb-40b3-a564-cf0b3a6f86bb/volumes" Nov 03 03:08:37.645043 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:37.645012 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:38.335062 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.335027 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn"] Nov 03 03:08:38.335725 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.335705 5463 cpu_manager.go:414] "containerMap: removing container" podUID="88e27111-aadb-40b3-a564-cf0b3a6f86bb" containerName="istio-proxy" Nov 03 03:08:38.335725 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.335725 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e27111-aadb-40b3-a564-cf0b3a6f86bb" containerName="istio-proxy" Nov 03 03:08:38.335951 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.335873 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e27111-aadb-40b3-a564-cf0b3a6f86bb" containerName="istio-proxy" Nov 03 03:08:38.340822 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.340797 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.394170 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.394148 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c7d758-openshift-default-dockercfg-kst7d\"" Nov 03 03:08:38.407166 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.407139 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/258c199f-3518-42ac-8962-d58bc543bc67-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.407324 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.407174 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.407405 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.407381 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzxqd\" (UniqueName: \"kubernetes.io/projected/258c199f-3518-42ac-8962-d58bc543bc67-kube-api-access-zzxqd\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.407465 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.407441 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.407537 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.407480 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.407599 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.407586 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/258c199f-3518-42ac-8962-d58bc543bc67-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.407661 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.407642 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.407717 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.407699 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/258c199f-3518-42ac-8962-d58bc543bc67-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.407774 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.407743 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.447878 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.447849 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn"] Nov 03 03:08:38.508821 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.508785 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/258c199f-3518-42ac-8962-d58bc543bc67-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.509008 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.508833 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.509008 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.508865 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/258c199f-3518-42ac-8962-d58bc543bc67-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.509008 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.508958 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.509181 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.509078 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/258c199f-3518-42ac-8962-d58bc543bc67-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.509181 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.509109 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.509324 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.509284 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzxqd\" (UniqueName: \"kubernetes.io/projected/258c199f-3518-42ac-8962-d58bc543bc67-kube-api-access-zzxqd\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.509383 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.509334 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.509383 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.509377 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.509499 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.509425 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.509702 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.509682 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.509785 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.509772 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.509847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.509775 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.510127 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.510104 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/258c199f-3518-42ac-8962-d58bc543bc67-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.511415 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.511392 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.511672 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.511652 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/258c199f-3518-42ac-8962-d58bc543bc67-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.567422 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.567392 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/258c199f-3518-42ac-8962-d58bc543bc67-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.569881 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.569858 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzxqd\" (UniqueName: \"kubernetes.io/projected/258c199f-3518-42ac-8962-d58bc543bc67-kube-api-access-zzxqd\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.657222 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.657160 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:38.997488 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:38.997340 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:39.090589 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:08:39.090523 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258c199f_3518_42ac_8962_d58bc543bc67.slice/crio-f78ac0fcda7f6cfc56a11ed25f2e014ef272f218cb36d27967d05b7aba30907e WatchSource:0}: Error finding container f78ac0fcda7f6cfc56a11ed25f2e014ef272f218cb36d27967d05b7aba30907e: Status 404 returned error can't find the container with id f78ac0fcda7f6cfc56a11ed25f2e014ef272f218cb36d27967d05b7aba30907e Nov 03 03:08:39.093389 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:39.093356 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:08:39.093494 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:39.093422 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:08:39.093494 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:39.093454 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:08:39.094872 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:39.094851 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn"] Nov 03 03:08:39.117514 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:39.117484 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" event={"ID":"258c199f-3518-42ac-8962-d58bc543bc67","Type":"ContainerStarted","Data":"f78ac0fcda7f6cfc56a11ed25f2e014ef272f218cb36d27967d05b7aba30907e"} Nov 03 03:08:39.180152 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:39.180123 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:40.126474 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:40.126443 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" event={"ID":"258c199f-3518-42ac-8962-d58bc543bc67","Type":"ContainerStarted","Data":"8bd750106a039418285bc56dc3afab85825bfe159e6f9bff475868b02db29ced"} Nov 03 03:08:40.274217 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:40.273640 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" podStartSLOduration=2.273620279 podStartE2EDuration="2.273620279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:08:40.268592975 +0000 UTC m=+4601.952702131" watchObservedRunningTime="2026-11-03 03:08:40.273620279 +0000 UTC m=+4601.957729435" Nov 03 03:08:40.671486 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:40.670473 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:40.680441 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:40.680417 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body= Nov 03 03:08:40.680631 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:40.680598 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" podUID="258c199f-3518-42ac-8962-d58bc543bc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" Nov 03 03:08:40.696343 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:40.696321 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7" Nov 03 03:08:40.736428 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:40.736404 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32df4d57-7e5b-4136-9e75-5c4e9290fd30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32df4d57-7e5b-4136-9e75-5c4e9290fd30-plugins-config-volume\") pod \"32df4d57-7e5b-4136-9e75-5c4e9290fd30\" (UID: \"32df4d57-7e5b-4136-9e75-5c4e9290fd30\") " Nov 03 03:08:40.736574 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:40.736455 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32df4d57-7e5b-4136-9e75-5c4e9290fd30-kube-api-access-m7c9p\" (UniqueName: \"kubernetes.io/projected/32df4d57-7e5b-4136-9e75-5c4e9290fd30-kube-api-access-m7c9p\") pod \"32df4d57-7e5b-4136-9e75-5c4e9290fd30\" (UID: \"32df4d57-7e5b-4136-9e75-5c4e9290fd30\") " Nov 03 03:08:40.736819 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:40.736796 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32df4d57-7e5b-4136-9e75-5c4e9290fd30-plugins-config-volume" pod "32df4d57-7e5b-4136-9e75-5c4e9290fd30" (UID: "32df4d57-7e5b-4136-9e75-5c4e9290fd30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:08:40.743716 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:40.743683 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32df4d57-7e5b-4136-9e75-5c4e9290fd30-kube-api-access-m7c9p" pod "32df4d57-7e5b-4136-9e75-5c4e9290fd30" (UID: "32df4d57-7e5b-4136-9e75-5c4e9290fd30"). InnerVolumeSpecName "kube-api-access-m7c9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:40.838604 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:40.838563 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32df4d57-7e5b-4136-9e75-5c4e9290fd30-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:40.838604 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:40.838605 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7c9p\" (UniqueName: \"kubernetes.io/projected/32df4d57-7e5b-4136-9e75-5c4e9290fd30-kube-api-access-m7c9p\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:40.885225 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:40.877757 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:41.092957 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:41.092884 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:41.134596 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:41.134568 5463 generic.go:356] "Generic (PLEG): container finished" podID="32df4d57-7e5b-4136-9e75-5c4e9290fd30" containerID="09860774bae4e8bca0e5fed3180b95a45c4fea28047e0801612cd93c1ff1e7e9" exitCode=0 Nov 03 03:08:41.135085 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:41.134627 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7" Nov 03 03:08:41.135085 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:41.134656 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7" event={"ID":"32df4d57-7e5b-4136-9e75-5c4e9290fd30","Type":"ContainerDied","Data":"09860774bae4e8bca0e5fed3180b95a45c4fea28047e0801612cd93c1ff1e7e9"} Nov 03 03:08:41.135085 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:41.134715 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7" event={"ID":"32df4d57-7e5b-4136-9e75-5c4e9290fd30","Type":"ContainerDied","Data":"f2e9f4bc29f0725ff844b6075ff4982542c31e16aba0984ed5ef4c7f3eefef67"} Nov 03 03:08:41.135085 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:41.134737 5463 scope.go:122] "RemoveContainer" containerID="09860774bae4e8bca0e5fed3180b95a45c4fea28047e0801612cd93c1ff1e7e9" Nov 03 03:08:41.145929 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:41.145913 5463 scope.go:122] "RemoveContainer" containerID="09860774bae4e8bca0e5fed3180b95a45c4fea28047e0801612cd93c1ff1e7e9" Nov 03 03:08:41.146233 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:08:41.146185 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09860774bae4e8bca0e5fed3180b95a45c4fea28047e0801612cd93c1ff1e7e9\": container with ID starting with 09860774bae4e8bca0e5fed3180b95a45c4fea28047e0801612cd93c1ff1e7e9 not found: ID does not exist" containerID="09860774bae4e8bca0e5fed3180b95a45c4fea28047e0801612cd93c1ff1e7e9" Nov 03 03:08:41.146308 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:41.146245 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09860774bae4e8bca0e5fed3180b95a45c4fea28047e0801612cd93c1ff1e7e9"} err="failed to get container status \"09860774bae4e8bca0e5fed3180b95a45c4fea28047e0801612cd93c1ff1e7e9\": rpc error: code = NotFound desc = could not find container \"09860774bae4e8bca0e5fed3180b95a45c4fea28047e0801612cd93c1ff1e7e9\": container with ID starting with 09860774bae4e8bca0e5fed3180b95a45c4fea28047e0801612cd93c1ff1e7e9 not found: ID does not exist" Nov 03 03:08:41.310905 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:41.310230 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7"] Nov 03 03:08:41.312811 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:41.312783 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c7d758-6fcdd7749f-ldpz7"] Nov 03 03:08:41.672672 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:41.672629 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body= Nov 03 03:08:41.672895 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:41.672696 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" podUID="258c199f-3518-42ac-8962-d58bc543bc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" Nov 03 03:08:42.273217 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:42.272534 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88" podUID="9b7b703f-6c31-4881-8e71-a23b27c3e6e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:08:42.443357 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:42.443325 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32df4d57-7e5b-4136-9e75-5c4e9290fd30" path="/var/lib/kubelet/pods/32df4d57-7e5b-4136-9e75-5c4e9290fd30/volumes" Nov 03 03:08:42.547292 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:42.547171 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:42.667215 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:42.659542 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body= Nov 03 03:08:42.667215 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:42.659604 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" podUID="258c199f-3518-42ac-8962-d58bc543bc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" Nov 03 03:08:43.008523 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:08:43.008483 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:08:43.663596 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:43.663561 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body= Nov 03 03:08:43.664347 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:43.664308 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" podUID="258c199f-3518-42ac-8962-d58bc543bc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" Nov 03 03:08:44.660514 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:44.660035 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body= Nov 03 03:08:44.660514 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:44.660107 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" podUID="258c199f-3518-42ac-8962-d58bc543bc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" Nov 03 03:08:45.657719 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:45.657679 5463 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body= Nov 03 03:08:45.658289 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:45.657755 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" podUID="258c199f-3518-42ac-8962-d58bc543bc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" Nov 03 03:08:45.941155 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:45.941051 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:46.564220 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:46.564160 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:46.663037 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:46.663006 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:46.663452 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:46.663401 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:46.664014 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:46.663990 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:08:47.349298 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:47.349265 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:51.013394 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:51.011023 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:51.020608 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:51.020589 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88" Nov 03 03:08:51.107387 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:51.107360 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b7b703f-6c31-4881-8e71-a23b27c3e6e7-kube-api-access-xsnfx\" (UniqueName: \"kubernetes.io/projected/9b7b703f-6c31-4881-8e71-a23b27c3e6e7-kube-api-access-xsnfx\") pod \"9b7b703f-6c31-4881-8e71-a23b27c3e6e7\" (UID: \"9b7b703f-6c31-4881-8e71-a23b27c3e6e7\") " Nov 03 03:08:51.107545 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:51.107431 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b7b703f-6c31-4881-8e71-a23b27c3e6e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7b703f-6c31-4881-8e71-a23b27c3e6e7-plugins-config-volume\") pod \"9b7b703f-6c31-4881-8e71-a23b27c3e6e7\" (UID: \"9b7b703f-6c31-4881-8e71-a23b27c3e6e7\") " Nov 03 03:08:51.107859 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:51.107830 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b7b703f-6c31-4881-8e71-a23b27c3e6e7-plugins-config-volume" pod "9b7b703f-6c31-4881-8e71-a23b27c3e6e7" (UID: "9b7b703f-6c31-4881-8e71-a23b27c3e6e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:08:51.108030 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:51.108015 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b7b703f-6c31-4881-8e71-a23b27c3e6e7-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:51.109572 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:51.109548 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b7b703f-6c31-4881-8e71-a23b27c3e6e7-kube-api-access-xsnfx" pod "9b7b703f-6c31-4881-8e71-a23b27c3e6e7" (UID: "9b7b703f-6c31-4881-8e71-a23b27c3e6e7"). InnerVolumeSpecName "kube-api-access-xsnfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:51.209467 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:51.209423 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsnfx\" (UniqueName: \"kubernetes.io/projected/9b7b703f-6c31-4881-8e71-a23b27c3e6e7-kube-api-access-xsnfx\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:08:51.227947 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:51.227913 5463 generic.go:356] "Generic (PLEG): container finished" podID="9b7b703f-6c31-4881-8e71-a23b27c3e6e7" containerID="cdef056da06131eb9232cb318b2df99b2afce31099086cb00998882adae253c5" exitCode=0 Nov 03 03:08:51.228117 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:51.227962 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88" Nov 03 03:08:51.228117 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:51.227991 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88" event={"ID":"9b7b703f-6c31-4881-8e71-a23b27c3e6e7","Type":"ContainerDied","Data":"cdef056da06131eb9232cb318b2df99b2afce31099086cb00998882adae253c5"} Nov 03 03:08:51.228117 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:51.228055 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88" event={"ID":"9b7b703f-6c31-4881-8e71-a23b27c3e6e7","Type":"ContainerDied","Data":"d582089b475d3ff3797a262fc9b59660ba5fec49bcf0458b733bc62269bef8b2"} Nov 03 03:08:51.228117 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:51.228085 5463 scope.go:122] "RemoveContainer" containerID="cdef056da06131eb9232cb318b2df99b2afce31099086cb00998882adae253c5" Nov 03 03:08:51.238174 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:51.238156 5463 scope.go:122] "RemoveContainer" containerID="cdef056da06131eb9232cb318b2df99b2afce31099086cb00998882adae253c5" Nov 03 03:08:51.238520 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:08:51.238497 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdef056da06131eb9232cb318b2df99b2afce31099086cb00998882adae253c5\": container with ID starting with cdef056da06131eb9232cb318b2df99b2afce31099086cb00998882adae253c5 not found: ID does not exist" containerID="cdef056da06131eb9232cb318b2df99b2afce31099086cb00998882adae253c5" Nov 03 03:08:51.238629 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:51.238525 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdef056da06131eb9232cb318b2df99b2afce31099086cb00998882adae253c5"} err="failed to get container status \"cdef056da06131eb9232cb318b2df99b2afce31099086cb00998882adae253c5\": rpc error: code = NotFound desc = could not find container \"cdef056da06131eb9232cb318b2df99b2afce31099086cb00998882adae253c5\": container with ID starting with cdef056da06131eb9232cb318b2df99b2afce31099086cb00998882adae253c5 not found: ID does not exist" Nov 03 03:08:51.376718 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:51.376620 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:51.414080 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:51.414053 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88"] Nov 03 03:08:51.435325 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:51.435299 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a0e72e-c6b668b4c-9xw88"] Nov 03 03:08:52.446020 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:52.445988 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b7b703f-6c31-4881-8e71-a23b27c3e6e7" path="/var/lib/kubelet/pods/9b7b703f-6c31-4881-8e71-a23b27c3e6e7/volumes" Nov 03 03:08:52.469729 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:52.469686 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:52.833686 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:52.833582 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:57.355594 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:57.355553 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:58.686016 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:08:58.685979 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:01.495424 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:01.495384 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:03.288169 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:03.288120 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:04.716219 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:04.716171 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:04.878670 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:04.878637 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:06.346671 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:06.346640 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:06.855996 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:06.855964 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv"] Nov 03 03:09:06.856231 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:06.856203 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" podUID="88462923-311f-4302-9503-b1dda3e244e2" containerName="istio-proxy" containerID="cri-o://32ab147d5ad68ccace0e681d63919a5809efd83fe3ff6968aca1976e586ae3da" gracePeriod=30 Nov 03 03:09:06.894201 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:06.894151 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:08.348446 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:08.348416 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:12.117145 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.117122 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 03:09:12.186905 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.186876 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-istio-data\") pod \"88462923-311f-4302-9503-b1dda3e244e2\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " Nov 03 03:09:12.187046 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.186906 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-credential-socket\") pod \"88462923-311f-4302-9503-b1dda3e244e2\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " Nov 03 03:09:12.187046 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.186931 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88462923-311f-4302-9503-b1dda3e244e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88462923-311f-4302-9503-b1dda3e244e2-istio-podinfo\") pod \"88462923-311f-4302-9503-b1dda3e244e2\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " Nov 03 03:09:12.187046 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.186962 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88462923-311f-4302-9503-b1dda3e244e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88462923-311f-4302-9503-b1dda3e244e2-istiod-ca-cert\") pod \"88462923-311f-4302-9503-b1dda3e244e2\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " Nov 03 03:09:12.187264 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.187061 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-workload-socket\") pod \"88462923-311f-4302-9503-b1dda3e244e2\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " Nov 03 03:09:12.187264 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.187094 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-workload-certs\") pod \"88462923-311f-4302-9503-b1dda3e244e2\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " Nov 03 03:09:12.187264 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.187165 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88462923-311f-4302-9503-b1dda3e244e2-istio-token\" (UniqueName: \"kubernetes.io/projected/88462923-311f-4302-9503-b1dda3e244e2-istio-token\") pod \"88462923-311f-4302-9503-b1dda3e244e2\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " Nov 03 03:09:12.187264 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.187220 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88462923-311f-4302-9503-b1dda3e244e2-kube-api-access-8l8n4\" (UniqueName: \"kubernetes.io/projected/88462923-311f-4302-9503-b1dda3e244e2-kube-api-access-8l8n4\") pod \"88462923-311f-4302-9503-b1dda3e244e2\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " Nov 03 03:09:12.187264 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.187171 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-istio-data" pod "88462923-311f-4302-9503-b1dda3e244e2" (UID: "88462923-311f-4302-9503-b1dda3e244e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:09:12.187264 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.187259 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-istio-envoy\") pod \"88462923-311f-4302-9503-b1dda3e244e2\" (UID: \"88462923-311f-4302-9503-b1dda3e244e2\") " Nov 03 03:09:12.187562 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.187265 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-credential-socket" pod "88462923-311f-4302-9503-b1dda3e244e2" (UID: "88462923-311f-4302-9503-b1dda3e244e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:09:12.187562 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.187375 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-workload-socket" pod "88462923-311f-4302-9503-b1dda3e244e2" (UID: "88462923-311f-4302-9503-b1dda3e244e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:09:12.187562 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.187474 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88462923-311f-4302-9503-b1dda3e244e2-istiod-ca-cert" pod "88462923-311f-4302-9503-b1dda3e244e2" (UID: "88462923-311f-4302-9503-b1dda3e244e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:09:12.187562 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.187481 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-workload-certs" pod "88462923-311f-4302-9503-b1dda3e244e2" (UID: "88462923-311f-4302-9503-b1dda3e244e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:09:12.187762 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.187701 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:12.187762 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.187720 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:12.187762 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.187731 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88462923-311f-4302-9503-b1dda3e244e2-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:12.187762 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.187739 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:12.187762 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.187749 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:12.189132 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.189110 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88462923-311f-4302-9503-b1dda3e244e2-istio-podinfo" pod "88462923-311f-4302-9503-b1dda3e244e2" (UID: "88462923-311f-4302-9503-b1dda3e244e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:09:12.189502 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.189481 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88462923-311f-4302-9503-b1dda3e244e2-kube-api-access-8l8n4" pod "88462923-311f-4302-9503-b1dda3e244e2" (UID: "88462923-311f-4302-9503-b1dda3e244e2"). InnerVolumeSpecName "kube-api-access-8l8n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:09:12.189744 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.189729 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-istio-envoy" pod "88462923-311f-4302-9503-b1dda3e244e2" (UID: "88462923-311f-4302-9503-b1dda3e244e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:09:12.189804 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.189783 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88462923-311f-4302-9503-b1dda3e244e2-istio-token" pod "88462923-311f-4302-9503-b1dda3e244e2" (UID: "88462923-311f-4302-9503-b1dda3e244e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:09:12.291801 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.291765 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88462923-311f-4302-9503-b1dda3e244e2-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:12.291801 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.291800 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l8n4\" (UniqueName: \"kubernetes.io/projected/88462923-311f-4302-9503-b1dda3e244e2-kube-api-access-8l8n4\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:12.292043 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.291815 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88462923-311f-4302-9503-b1dda3e244e2-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:12.292043 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.291829 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88462923-311f-4302-9503-b1dda3e244e2-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:12.439970 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.439892 5463 generic.go:356] "Generic (PLEG): container finished" podID="88462923-311f-4302-9503-b1dda3e244e2" containerID="32ab147d5ad68ccace0e681d63919a5809efd83fe3ff6968aca1976e586ae3da" exitCode=0 Nov 03 03:09:12.440136 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.440034 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" Nov 03 03:09:12.443468 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.443444 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" event={"ID":"88462923-311f-4302-9503-b1dda3e244e2","Type":"ContainerDied","Data":"32ab147d5ad68ccace0e681d63919a5809efd83fe3ff6968aca1976e586ae3da"} Nov 03 03:09:12.443600 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.443484 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv" event={"ID":"88462923-311f-4302-9503-b1dda3e244e2","Type":"ContainerDied","Data":"1b92c78ca485c5cb0a648dd77fc5ebccd41a9308a8e37e3e65b9fe9917597633"} Nov 03 03:09:12.443600 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.443506 5463 scope.go:122] "RemoveContainer" containerID="32ab147d5ad68ccace0e681d63919a5809efd83fe3ff6968aca1976e586ae3da" Nov 03 03:09:12.454169 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.454154 5463 scope.go:122] "RemoveContainer" containerID="fc5953001997a0d8392b007654bb435c0fbd39fad5bdfa6f8ce6b6f723276eae" Nov 03 03:09:12.463097 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.463079 5463 scope.go:122] "RemoveContainer" containerID="32ab147d5ad68ccace0e681d63919a5809efd83fe3ff6968aca1976e586ae3da" Nov 03 03:09:12.463394 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:09:12.463372 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ab147d5ad68ccace0e681d63919a5809efd83fe3ff6968aca1976e586ae3da\": container with ID starting with 32ab147d5ad68ccace0e681d63919a5809efd83fe3ff6968aca1976e586ae3da not found: ID does not exist" containerID="32ab147d5ad68ccace0e681d63919a5809efd83fe3ff6968aca1976e586ae3da" Nov 03 03:09:12.463450 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.463403 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ab147d5ad68ccace0e681d63919a5809efd83fe3ff6968aca1976e586ae3da"} err="failed to get container status \"32ab147d5ad68ccace0e681d63919a5809efd83fe3ff6968aca1976e586ae3da\": rpc error: code = NotFound desc = could not find container \"32ab147d5ad68ccace0e681d63919a5809efd83fe3ff6968aca1976e586ae3da\": container with ID starting with 32ab147d5ad68ccace0e681d63919a5809efd83fe3ff6968aca1976e586ae3da not found: ID does not exist" Nov 03 03:09:12.463450 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.463422 5463 scope.go:122] "RemoveContainer" containerID="fc5953001997a0d8392b007654bb435c0fbd39fad5bdfa6f8ce6b6f723276eae" Nov 03 03:09:12.463658 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:09:12.463641 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc5953001997a0d8392b007654bb435c0fbd39fad5bdfa6f8ce6b6f723276eae\": container with ID starting with fc5953001997a0d8392b007654bb435c0fbd39fad5bdfa6f8ce6b6f723276eae not found: ID does not exist" containerID="fc5953001997a0d8392b007654bb435c0fbd39fad5bdfa6f8ce6b6f723276eae" Nov 03 03:09:12.463704 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.463664 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc5953001997a0d8392b007654bb435c0fbd39fad5bdfa6f8ce6b6f723276eae"} err="failed to get container status \"fc5953001997a0d8392b007654bb435c0fbd39fad5bdfa6f8ce6b6f723276eae\": rpc error: code = NotFound desc = could not find container \"fc5953001997a0d8392b007654bb435c0fbd39fad5bdfa6f8ce6b6f723276eae\": container with ID starting with fc5953001997a0d8392b007654bb435c0fbd39fad5bdfa6f8ce6b6f723276eae not found: ID does not exist" Nov 03 03:09:12.611690 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.611656 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv"] Nov 03 03:09:12.617185 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:12.617161 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-2dvjv"] Nov 03 03:09:13.006930 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:09:13.006890 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:09:13.250573 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:13.250534 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:14.442935 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:14.442901 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88462923-311f-4302-9503-b1dda3e244e2" path="/var/lib/kubelet/pods/88462923-311f-4302-9503-b1dda3e244e2/volumes" Nov 03 03:09:15.729441 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:15.729401 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:17.347768 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:17.347730 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:19.141288 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:19.141250 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:21.200463 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:21.200419 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:21.408648 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:21.408615 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:22.794291 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:22.794251 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:23.273962 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:23.273926 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn"] Nov 03 03:09:23.274264 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:23.274239 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" podUID="258c199f-3518-42ac-8962-d58bc543bc67" containerName="istio-proxy" containerID="cri-o://8bd750106a039418285bc56dc3afab85825bfe159e6f9bff475868b02db29ced" gracePeriod=30 Nov 03 03:09:23.294653 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:23.294633 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:24.141077 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:24.141041 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:28.542264 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.542234 5463 generic.go:356] "Generic (PLEG): container finished" podID="258c199f-3518-42ac-8962-d58bc543bc67" containerID="8bd750106a039418285bc56dc3afab85825bfe159e6f9bff475868b02db29ced" exitCode=0 Nov 03 03:09:28.542576 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.542304 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" event={"ID":"258c199f-3518-42ac-8962-d58bc543bc67","Type":"ContainerDied","Data":"8bd750106a039418285bc56dc3afab85825bfe159e6f9bff475868b02db29ced"} Nov 03 03:09:28.542576 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.542363 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" event={"ID":"258c199f-3518-42ac-8962-d58bc543bc67","Type":"ContainerDied","Data":"f78ac0fcda7f6cfc56a11ed25f2e014ef272f218cb36d27967d05b7aba30907e"} Nov 03 03:09:28.542576 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.542407 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f78ac0fcda7f6cfc56a11ed25f2e014ef272f218cb36d27967d05b7aba30907e" Nov 03 03:09:28.548094 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.547998 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:28.559289 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.559266 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:09:28.690176 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.690140 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/258c199f-3518-42ac-8962-d58bc543bc67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/258c199f-3518-42ac-8962-d58bc543bc67-istio-podinfo\") pod \"258c199f-3518-42ac-8962-d58bc543bc67\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " Nov 03 03:09:28.690176 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.690174 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/258c199f-3518-42ac-8962-d58bc543bc67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/258c199f-3518-42ac-8962-d58bc543bc67-istiod-ca-cert\") pod \"258c199f-3518-42ac-8962-d58bc543bc67\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " Nov 03 03:09:28.690437 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.690236 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-workload-socket\") pod \"258c199f-3518-42ac-8962-d58bc543bc67\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " Nov 03 03:09:28.690437 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.690279 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/258c199f-3518-42ac-8962-d58bc543bc67-kube-api-access-zzxqd\" (UniqueName: \"kubernetes.io/projected/258c199f-3518-42ac-8962-d58bc543bc67-kube-api-access-zzxqd\") pod \"258c199f-3518-42ac-8962-d58bc543bc67\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " Nov 03 03:09:28.690437 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.690313 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-istio-data\") pod \"258c199f-3518-42ac-8962-d58bc543bc67\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " Nov 03 03:09:28.690437 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.690368 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-istio-envoy\") pod \"258c199f-3518-42ac-8962-d58bc543bc67\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " Nov 03 03:09:28.690667 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.690554 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-credential-socket\") pod \"258c199f-3518-42ac-8962-d58bc543bc67\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " Nov 03 03:09:28.690667 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.690623 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-workload-certs\") pod \"258c199f-3518-42ac-8962-d58bc543bc67\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " Nov 03 03:09:28.690792 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.690739 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/258c199f-3518-42ac-8962-d58bc543bc67-istio-token\" (UniqueName: \"kubernetes.io/projected/258c199f-3518-42ac-8962-d58bc543bc67-istio-token\") pod \"258c199f-3518-42ac-8962-d58bc543bc67\" (UID: \"258c199f-3518-42ac-8962-d58bc543bc67\") " Nov 03 03:09:28.690851 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.690820 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-workload-socket" pod "258c199f-3518-42ac-8962-d58bc543bc67" (UID: "258c199f-3518-42ac-8962-d58bc543bc67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:09:28.691357 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.691174 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/258c199f-3518-42ac-8962-d58bc543bc67-istiod-ca-cert" pod "258c199f-3518-42ac-8962-d58bc543bc67" (UID: "258c199f-3518-42ac-8962-d58bc543bc67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:09:28.691357 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.691275 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:28.691627 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.691590 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-credential-socket" pod "258c199f-3518-42ac-8962-d58bc543bc67" (UID: "258c199f-3518-42ac-8962-d58bc543bc67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:09:28.691854 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.691831 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-workload-certs" pod "258c199f-3518-42ac-8962-d58bc543bc67" (UID: "258c199f-3518-42ac-8962-d58bc543bc67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:09:28.691992 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.691947 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-istio-data" pod "258c199f-3518-42ac-8962-d58bc543bc67" (UID: "258c199f-3518-42ac-8962-d58bc543bc67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:09:28.692994 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.692969 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/258c199f-3518-42ac-8962-d58bc543bc67-istio-podinfo" pod "258c199f-3518-42ac-8962-d58bc543bc67" (UID: "258c199f-3518-42ac-8962-d58bc543bc67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:09:28.694125 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.694088 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-istio-envoy" pod "258c199f-3518-42ac-8962-d58bc543bc67" (UID: "258c199f-3518-42ac-8962-d58bc543bc67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:09:28.694318 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.694290 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258c199f-3518-42ac-8962-d58bc543bc67-istio-token" pod "258c199f-3518-42ac-8962-d58bc543bc67" (UID: "258c199f-3518-42ac-8962-d58bc543bc67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:09:28.694930 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.694904 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258c199f-3518-42ac-8962-d58bc543bc67-kube-api-access-zzxqd" pod "258c199f-3518-42ac-8962-d58bc543bc67" (UID: "258c199f-3518-42ac-8962-d58bc543bc67"). InnerVolumeSpecName "kube-api-access-zzxqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:09:28.792241 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.792155 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/258c199f-3518-42ac-8962-d58bc543bc67-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:28.792241 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.792221 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/258c199f-3518-42ac-8962-d58bc543bc67-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:28.792241 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.792238 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/258c199f-3518-42ac-8962-d58bc543bc67-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:28.792471 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.792252 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzxqd\" (UniqueName: \"kubernetes.io/projected/258c199f-3518-42ac-8962-d58bc543bc67-kube-api-access-zzxqd\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:28.792471 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.792269 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:28.792471 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.792283 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:28.792471 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.792296 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:28.792471 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:28.792310 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/258c199f-3518-42ac-8962-d58bc543bc67-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:29.100455 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:29.100352 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:29.549677 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:29.549645 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn" Nov 03 03:09:29.767864 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:29.767830 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn"] Nov 03 03:09:29.771843 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:29.771817 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-xlxqn"] Nov 03 03:09:30.442537 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:30.442505 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="258c199f-3518-42ac-8962-d58bc543bc67" path="/var/lib/kubelet/pods/258c199f-3518-42ac-8962-d58bc543bc67/volumes" Nov 03 03:09:33.138085 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:33.138052 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:33.844041 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:33.844005 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:38.649230 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:38.649180 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:38.857619 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:38.857577 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:39.024818 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:39.024739 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:40.835369 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:40.835336 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:41.011150 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:41.011114 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:41.351665 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:41.351631 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:41.582751 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:41.582716 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:43.007831 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:09:43.007796 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:09:43.447625 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:43.447594 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:48.216223 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:48.216160 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:52.860691 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:52.860656 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:57.665701 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:09:57.665658 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:02.429720 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:10:02.429681 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:07.188564 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:10:07.188531 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:11.943082 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:10:11.943002 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:13.009772 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:10:13.009738 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:10:16.776868 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:10:16.776811 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:21.469692 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:10:21.469655 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:26.240893 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:10:26.240854 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:28.854545 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:10:28.854511 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:30.929695 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:10:30.929660 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:31.871214 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:10:31.871150 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:33.238604 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:10:33.238569 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:33.707065 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:10:33.707037 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:35.722809 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:10:35.722768 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:38.731688 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:10:38.731651 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:40.578756 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:10:40.578717 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:42.899711 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:10:42.899673 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:43.009761 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:10:43.009723 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:10:43.044948 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:10:43.044921 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:45.343020 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:10:45.342982 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:50.136421 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:10:50.136383 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:54.847764 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:10:54.847727 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:59.626778 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:10:59.626743 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:04.351182 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:04.351143 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:09.135635 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:09.135592 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:13.007631 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:11:13.007597 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:11:13.971927 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:13.971856 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:14.817177 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:14.817146 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:14.974490 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:14.974448 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:16.188593 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:16.188560 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:16.606408 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:16.606312 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:16.778834 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:16.778797 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:16.936589 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:16.936557 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:18.947565 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:18.947534 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:21.822212 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:21.822147 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:22.493398 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:22.493364 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:22.705018 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:22.704979 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:23.830705 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:23.830668 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:26.179644 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:26.179606 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:28.624132 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:28.624092 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:29.163276 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:29.163244 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:29.364700 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:29.364665 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:33.424416 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:33.424373 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:37.548370 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:37.548327 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:38.808524 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:38.808492 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:39.325938 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:39.325905 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:43.009119 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:11:43.009079 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:11:44.078770 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:44.078721 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:45.062221 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:45.062129 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:45.329101 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:45.328994 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:46.658106 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:46.658073 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:47.304446 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:47.304414 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp"] Nov 03 03:11:47.304685 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:47.304660 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" podUID="5ddcf4ff-6e4f-45cd-81a8-8b283429291a" containerName="istio-proxy" containerID="cri-o://ef5a7c160b9241132bb442aa14c975134886b8f3263669e78c7ef35c43d8b634" gracePeriod=30 Nov 03 03:11:47.317641 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:47.317612 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:48.111371 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:48.111336 5463 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:11:48.111771 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:48.111406 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" podUID="5ddcf4ff-6e4f-45cd-81a8-8b283429291a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:11:49.139108 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:49.139070 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:49.289089 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:49.289060 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:52.552767 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.552728 5463 generic.go:356] "Generic (PLEG): container finished" podID="5ddcf4ff-6e4f-45cd-81a8-8b283429291a" containerID="ef5a7c160b9241132bb442aa14c975134886b8f3263669e78c7ef35c43d8b634" exitCode=0 Nov 03 03:11:52.553168 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.552806 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" event={"ID":"5ddcf4ff-6e4f-45cd-81a8-8b283429291a","Type":"ContainerDied","Data":"ef5a7c160b9241132bb442aa14c975134886b8f3263669e78c7ef35c43d8b634"} Nov 03 03:11:52.553168 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.552885 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" event={"ID":"5ddcf4ff-6e4f-45cd-81a8-8b283429291a","Type":"ContainerDied","Data":"87903f50bf49cabecece6e47c78efb206582fb0eba06ff38e81d5e2d51fbef2a"} Nov 03 03:11:52.553168 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.552912 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87903f50bf49cabecece6e47c78efb206582fb0eba06ff38e81d5e2d51fbef2a" Nov 03 03:11:52.553168 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.552932 5463 scope.go:122] "RemoveContainer" containerID="8f57f4bda498318d5b27f65f2842ab61418b7f4cb5577faa26326b5bc9ee2154" Nov 03 03:11:52.570452 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.570408 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 03:11:52.659850 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.659826 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-podinfo\") pod \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " Nov 03 03:11:52.659983 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.659862 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-data\") pod \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " Nov 03 03:11:52.659983 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.659936 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-workload-certs\") pod \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " Nov 03 03:11:52.659983 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.659963 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-envoy\") pod \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " Nov 03 03:11:52.660168 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.660083 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-kube-api-access-k55xx\" (UniqueName: \"kubernetes.io/projected/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-kube-api-access-k55xx\") pod \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " Nov 03 03:11:52.660168 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.660137 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-token\" (UniqueName: \"kubernetes.io/projected/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-token\") pod \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " Nov 03 03:11:52.660318 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.660166 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-workload-socket\") pod \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " Nov 03 03:11:52.660318 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.660258 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-data" pod "5ddcf4ff-6e4f-45cd-81a8-8b283429291a" (UID: "5ddcf4ff-6e4f-45cd-81a8-8b283429291a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:11:52.660318 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.660264 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-credential-socket\") pod \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " Nov 03 03:11:52.660318 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.660272 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-workload-certs" pod "5ddcf4ff-6e4f-45cd-81a8-8b283429291a" (UID: "5ddcf4ff-6e4f-45cd-81a8-8b283429291a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:11:52.660534 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.660347 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istiod-ca-cert\") pod \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\" (UID: \"5ddcf4ff-6e4f-45cd-81a8-8b283429291a\") " Nov 03 03:11:52.660534 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.660488 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-credential-socket" pod "5ddcf4ff-6e4f-45cd-81a8-8b283429291a" (UID: "5ddcf4ff-6e4f-45cd-81a8-8b283429291a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:11:52.660534 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.660500 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-workload-socket" pod "5ddcf4ff-6e4f-45cd-81a8-8b283429291a" (UID: "5ddcf4ff-6e4f-45cd-81a8-8b283429291a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:11:52.660808 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.660752 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:52.660808 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.660775 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:52.660808 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.660785 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:52.660808 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.660793 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:52.661019 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.660841 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istiod-ca-cert" pod "5ddcf4ff-6e4f-45cd-81a8-8b283429291a" (UID: "5ddcf4ff-6e4f-45cd-81a8-8b283429291a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:11:52.662413 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.662385 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-podinfo" pod "5ddcf4ff-6e4f-45cd-81a8-8b283429291a" (UID: "5ddcf4ff-6e4f-45cd-81a8-8b283429291a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:11:52.662736 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.662712 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-kube-api-access-k55xx" pod "5ddcf4ff-6e4f-45cd-81a8-8b283429291a" (UID: "5ddcf4ff-6e4f-45cd-81a8-8b283429291a"). InnerVolumeSpecName "kube-api-access-k55xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:11:52.662736 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.662724 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-token" pod "5ddcf4ff-6e4f-45cd-81a8-8b283429291a" (UID: "5ddcf4ff-6e4f-45cd-81a8-8b283429291a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:11:52.662851 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.662727 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-envoy" pod "5ddcf4ff-6e4f-45cd-81a8-8b283429291a" (UID: "5ddcf4ff-6e4f-45cd-81a8-8b283429291a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:11:52.761628 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.761602 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k55xx\" (UniqueName: \"kubernetes.io/projected/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-kube-api-access-k55xx\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:52.761769 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.761632 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:52.761769 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.761647 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:52.761769 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.761660 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:52.761769 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:52.761674 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ddcf4ff-6e4f-45cd-81a8-8b283429291a-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:53.560214 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:53.560176 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp" Nov 03 03:11:53.794328 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:53.794292 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:53.847499 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:53.847398 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp"] Nov 03 03:11:53.853012 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:53.852978 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9aebe8-openshift-default-5bdb5554c5-jkzcp"] Nov 03 03:11:53.934560 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:53.934524 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:54.233830 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:54.233799 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:54.425876 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:54.425842 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:54.443296 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:54.443265 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ddcf4ff-6e4f-45cd-81a8-8b283429291a" path="/var/lib/kubelet/pods/5ddcf4ff-6e4f-45cd-81a8-8b283429291a/volumes" Nov 03 03:11:56.979950 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:56.979917 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65"] Nov 03 03:11:57.008672 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:57.008612 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:57.518650 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:57.518617 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:59.346612 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:59.346563 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:59.955161 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:11:59.955133 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:02.622246 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:02.622215 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8"] Nov 03 03:12:02.732390 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:02.732356 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns"] Nov 03 03:12:03.011263 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:03.011228 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:03.174240 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:03.174185 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:03.310435 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:03.310351 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 03:12:03.320425 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:03.320404 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 03:12:03.656286 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:03.656256 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:04.290510 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:04.290474 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:05.948392 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:05.948358 5463 scope.go:122] "RemoveContainer" containerID="ef5a7c160b9241132bb442aa14c975134886b8f3263669e78c7ef35c43d8b634" Nov 03 03:12:07.622532 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:07.622488 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8" podUID="da6e7a0f-9fd7-4115-9c62-caed63f180de" containerName="payload-processing" containerID="cri-o://a67c47e441a2d717ba1dcd75ca3ed7911d700e8c1f0c080a2028e1b56370804d" gracePeriod=25 Nov 03 03:12:07.732704 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:07.732670 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns" podUID="d683621e-ee05-4efe-9d28-49d1f085d135" containerName="payload-pre-processing" containerID="cri-o://b409b192f53cda255baf81a70689f4c79362d44c78d9e2e563611fc57e524df3" gracePeriod=25 Nov 03 03:12:08.652667 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:08.652625 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:09.307157 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:09.307119 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:10.446418 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:10.446384 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:12.001044 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.001003 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" podUID="1ee1477e-0bf0-428b-ac3f-03e5f217116f" containerName="main" containerID="cri-o://8bfa16eed80097a3c309f3f43ad7b2c6b18d09d1eb64af541b6f333f94ce6d80" gracePeriod=45 Nov 03 03:12:12.279392 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.279365 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:12:12.418914 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.418873 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-model-cache\") pod \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " Nov 03 03:12:12.419117 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.418939 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-tmp-dir\") pod \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " Nov 03 03:12:12.419117 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.419018 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ee1477e-0bf0-428b-ac3f-03e5f217116f-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee1477e-0bf0-428b-ac3f-03e5f217116f-tls-certs\") pod \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " Nov 03 03:12:12.419117 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.419051 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ee1477e-0bf0-428b-ac3f-03e5f217116f-kube-api-access-vtqwd\" (UniqueName: \"kubernetes.io/projected/1ee1477e-0bf0-428b-ac3f-03e5f217116f-kube-api-access-vtqwd\") pod \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " Nov 03 03:12:12.419356 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.419171 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-dshm\") pod \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " Nov 03 03:12:12.419356 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.419230 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-model-cache" pod "1ee1477e-0bf0-428b-ac3f-03e5f217116f" (UID: "1ee1477e-0bf0-428b-ac3f-03e5f217116f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:12.419356 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.419235 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-tmp-dir" pod "1ee1477e-0bf0-428b-ac3f-03e5f217116f" (UID: "1ee1477e-0bf0-428b-ac3f-03e5f217116f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:12.419356 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.419243 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-home\" (UniqueName: \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-home\") pod \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\" (UID: \"1ee1477e-0bf0-428b-ac3f-03e5f217116f\") " Nov 03 03:12:12.419972 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.419762 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-home" pod "1ee1477e-0bf0-428b-ac3f-03e5f217116f" (UID: "1ee1477e-0bf0-428b-ac3f-03e5f217116f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:12.419972 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.419883 5463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-home\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:12.419972 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.419906 5463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-model-cache\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:12.420280 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.419920 5463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-tmp-dir\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:12.421967 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.421915 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ee1477e-0bf0-428b-ac3f-03e5f217116f-tls-certs" pod "1ee1477e-0bf0-428b-ac3f-03e5f217116f" (UID: "1ee1477e-0bf0-428b-ac3f-03e5f217116f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:12:12.422162 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.422009 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-dshm" pod "1ee1477e-0bf0-428b-ac3f-03e5f217116f" (UID: "1ee1477e-0bf0-428b-ac3f-03e5f217116f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:12.422162 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.422124 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee1477e-0bf0-428b-ac3f-03e5f217116f-kube-api-access-vtqwd" pod "1ee1477e-0bf0-428b-ac3f-03e5f217116f" (UID: "1ee1477e-0bf0-428b-ac3f-03e5f217116f"). InnerVolumeSpecName "kube-api-access-vtqwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:12.521112 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.521032 5463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee1477e-0bf0-428b-ac3f-03e5f217116f-dshm\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:12.521112 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.521062 5463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee1477e-0bf0-428b-ac3f-03e5f217116f-tls-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:12.521112 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.521072 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtqwd\" (UniqueName: \"kubernetes.io/projected/1ee1477e-0bf0-428b-ac3f-03e5f217116f-kube-api-access-vtqwd\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:12.696892 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.696856 5463 generic.go:356] "Generic (PLEG): container finished" podID="1ee1477e-0bf0-428b-ac3f-03e5f217116f" containerID="8bfa16eed80097a3c309f3f43ad7b2c6b18d09d1eb64af541b6f333f94ce6d80" exitCode=0 Nov 03 03:12:12.697074 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.696904 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" Nov 03 03:12:12.697074 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.696942 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" event={"ID":"1ee1477e-0bf0-428b-ac3f-03e5f217116f","Type":"ContainerDied","Data":"8bfa16eed80097a3c309f3f43ad7b2c6b18d09d1eb64af541b6f333f94ce6d80"} Nov 03 03:12:12.697074 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.696985 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65" event={"ID":"1ee1477e-0bf0-428b-ac3f-03e5f217116f","Type":"ContainerDied","Data":"b4ddfc564d315de80d0de3545814b0274a834e475daae5edfd30d297810a86d8"} Nov 03 03:12:12.697074 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.697003 5463 scope.go:122] "RemoveContainer" containerID="8bfa16eed80097a3c309f3f43ad7b2c6b18d09d1eb64af541b6f333f94ce6d80" Nov 03 03:12:12.706960 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.706943 5463 scope.go:122] "RemoveContainer" containerID="8bfa16eed80097a3c309f3f43ad7b2c6b18d09d1eb64af541b6f333f94ce6d80" Nov 03 03:12:12.707236 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:12:12.707217 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bfa16eed80097a3c309f3f43ad7b2c6b18d09d1eb64af541b6f333f94ce6d80\": container with ID starting with 8bfa16eed80097a3c309f3f43ad7b2c6b18d09d1eb64af541b6f333f94ce6d80 not found: ID does not exist" containerID="8bfa16eed80097a3c309f3f43ad7b2c6b18d09d1eb64af541b6f333f94ce6d80" Nov 03 03:12:12.707310 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.707249 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bfa16eed80097a3c309f3f43ad7b2c6b18d09d1eb64af541b6f333f94ce6d80"} err="failed to get container status \"8bfa16eed80097a3c309f3f43ad7b2c6b18d09d1eb64af541b6f333f94ce6d80\": rpc error: code = NotFound desc = could not find container \"8bfa16eed80097a3c309f3f43ad7b2c6b18d09d1eb64af541b6f333f94ce6d80\": container with ID starting with 8bfa16eed80097a3c309f3f43ad7b2c6b18d09d1eb64af541b6f333f94ce6d80 not found: ID does not exist" Nov 03 03:12:12.863239 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.863155 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65"] Nov 03 03:12:12.866444 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:12.866421 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-78b5c5/sub-test-model-78b5c5-kserve-697d6785c-xgv65"] Nov 03 03:12:13.007119 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:12:13.007077 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:12:14.443208 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:14.443158 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ee1477e-0bf0-428b-ac3f-03e5f217116f" path="/var/lib/kubelet/pods/1ee1477e-0bf0-428b-ac3f-03e5f217116f/volumes" Nov 03 03:12:15.039454 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:15.039422 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:15.206387 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:15.206347 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:15.404439 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:15.404403 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:16.646957 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:16.646913 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:16.813087 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:16.813041 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns" podUID="d683621e-ee05-4efe-9d28-49d1f085d135" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:12:16.813302 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:16.813163 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8" podUID="da6e7a0f-9fd7-4115-9c62-caed63f180de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:12:17.231812 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:17.231780 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:18.042369 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:18.042337 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:20.524420 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:20.524388 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:23.641131 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:23.641096 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv"] Nov 03 03:12:24.063554 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:24.063513 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:24.227152 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:24.227121 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:24.592210 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:24.592151 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:24.750113 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:24.750081 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:26.730138 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:26.730104 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:26.813284 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:26.813243 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8" podUID="da6e7a0f-9fd7-4115-9c62-caed63f180de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:12:26.813284 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:26.813266 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns" podUID="d683621e-ee05-4efe-9d28-49d1f085d135" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:12:27.804829 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:27.801728 5463 generic.go:356] "Generic (PLEG): container finished" podID="da6e7a0f-9fd7-4115-9c62-caed63f180de" containerID="a67c47e441a2d717ba1dcd75ca3ed7911d700e8c1f0c080a2028e1b56370804d" exitCode=0 Nov 03 03:12:27.804829 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:27.801767 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8" event={"ID":"da6e7a0f-9fd7-4115-9c62-caed63f180de","Type":"ContainerDied","Data":"a67c47e441a2d717ba1dcd75ca3ed7911d700e8c1f0c080a2028e1b56370804d"} Nov 03 03:12:27.903100 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:27.903069 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8" Nov 03 03:12:28.012814 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.012791 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns" Nov 03 03:12:28.014867 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.014849 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da6e7a0f-9fd7-4115-9c62-caed63f180de-kube-api-access-f4vsq\" (UniqueName: \"kubernetes.io/projected/da6e7a0f-9fd7-4115-9c62-caed63f180de-kube-api-access-f4vsq\") pod \"da6e7a0f-9fd7-4115-9c62-caed63f180de\" (UID: \"da6e7a0f-9fd7-4115-9c62-caed63f180de\") " Nov 03 03:12:28.014945 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.014930 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da6e7a0f-9fd7-4115-9c62-caed63f180de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6e7a0f-9fd7-4115-9c62-caed63f180de-plugins-config-volume\") pod \"da6e7a0f-9fd7-4115-9c62-caed63f180de\" (UID: \"da6e7a0f-9fd7-4115-9c62-caed63f180de\") " Nov 03 03:12:28.015364 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.015336 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6e7a0f-9fd7-4115-9c62-caed63f180de-plugins-config-volume" pod "da6e7a0f-9fd7-4115-9c62-caed63f180de" (UID: "da6e7a0f-9fd7-4115-9c62-caed63f180de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:12:28.015824 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.015803 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6e7a0f-9fd7-4115-9c62-caed63f180de-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:28.017610 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.017583 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6e7a0f-9fd7-4115-9c62-caed63f180de-kube-api-access-f4vsq" pod "da6e7a0f-9fd7-4115-9c62-caed63f180de" (UID: "da6e7a0f-9fd7-4115-9c62-caed63f180de"). InnerVolumeSpecName "kube-api-access-f4vsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:28.116648 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.116618 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d683621e-ee05-4efe-9d28-49d1f085d135-kube-api-access-9plnx\" (UniqueName: \"kubernetes.io/projected/d683621e-ee05-4efe-9d28-49d1f085d135-kube-api-access-9plnx\") pod \"d683621e-ee05-4efe-9d28-49d1f085d135\" (UID: \"d683621e-ee05-4efe-9d28-49d1f085d135\") " Nov 03 03:12:28.116827 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.116660 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d683621e-ee05-4efe-9d28-49d1f085d135-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d683621e-ee05-4efe-9d28-49d1f085d135-plugins-config-volume\") pod \"d683621e-ee05-4efe-9d28-49d1f085d135\" (UID: \"d683621e-ee05-4efe-9d28-49d1f085d135\") " Nov 03 03:12:28.117056 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.117030 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d683621e-ee05-4efe-9d28-49d1f085d135-plugins-config-volume" pod "d683621e-ee05-4efe-9d28-49d1f085d135" (UID: "d683621e-ee05-4efe-9d28-49d1f085d135"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:12:28.117111 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.117065 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4vsq\" (UniqueName: \"kubernetes.io/projected/da6e7a0f-9fd7-4115-9c62-caed63f180de-kube-api-access-f4vsq\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:28.118750 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.118731 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d683621e-ee05-4efe-9d28-49d1f085d135-kube-api-access-9plnx" pod "d683621e-ee05-4efe-9d28-49d1f085d135" (UID: "d683621e-ee05-4efe-9d28-49d1f085d135"). InnerVolumeSpecName "kube-api-access-9plnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:28.218236 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.218204 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9plnx\" (UniqueName: \"kubernetes.io/projected/d683621e-ee05-4efe-9d28-49d1f085d135-kube-api-access-9plnx\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:28.218236 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.218234 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d683621e-ee05-4efe-9d28-49d1f085d135-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:28.641639 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.641592 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv" podUID="ed76fc49-8f0a-4a64-a273-01cb7a2d612b" containerName="payload-pre-processing" containerID="cri-o://936a662c6ae831a69413aaeedfa9309906acc3a4bd8b88fefc2a6dbe6436d34e" gracePeriod=25 Nov 03 03:12:28.809497 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.809472 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8" Nov 03 03:12:28.809973 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.809499 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8" event={"ID":"da6e7a0f-9fd7-4115-9c62-caed63f180de","Type":"ContainerDied","Data":"50eaead815e755bcecc0bf1af23ec3159e6d692a1b3dc591eb0f4f5c3df77b37"} Nov 03 03:12:28.809973 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.809562 5463 scope.go:122] "RemoveContainer" containerID="a67c47e441a2d717ba1dcd75ca3ed7911d700e8c1f0c080a2028e1b56370804d" Nov 03 03:12:28.810927 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.810839 5463 generic.go:356] "Generic (PLEG): container finished" podID="d683621e-ee05-4efe-9d28-49d1f085d135" containerID="b409b192f53cda255baf81a70689f4c79362d44c78d9e2e563611fc57e524df3" exitCode=0 Nov 03 03:12:28.810927 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.810873 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns" event={"ID":"d683621e-ee05-4efe-9d28-49d1f085d135","Type":"ContainerDied","Data":"b409b192f53cda255baf81a70689f4c79362d44c78d9e2e563611fc57e524df3"} Nov 03 03:12:28.810927 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.810890 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns" Nov 03 03:12:28.811130 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.810930 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns" event={"ID":"d683621e-ee05-4efe-9d28-49d1f085d135","Type":"ContainerDied","Data":"910e33ff49ae8e4ee1c2c404dd02e185a7a170e9a7f394192afbd41ed25392b6"} Nov 03 03:12:28.819071 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.819050 5463 scope.go:122] "RemoveContainer" containerID="b409b192f53cda255baf81a70689f4c79362d44c78d9e2e563611fc57e524df3" Nov 03 03:12:28.827401 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.827385 5463 scope.go:122] "RemoveContainer" containerID="b409b192f53cda255baf81a70689f4c79362d44c78d9e2e563611fc57e524df3" Nov 03 03:12:28.827649 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:12:28.827633 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b409b192f53cda255baf81a70689f4c79362d44c78d9e2e563611fc57e524df3\": container with ID starting with b409b192f53cda255baf81a70689f4c79362d44c78d9e2e563611fc57e524df3 not found: ID does not exist" containerID="b409b192f53cda255baf81a70689f4c79362d44c78d9e2e563611fc57e524df3" Nov 03 03:12:28.827707 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.827655 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b409b192f53cda255baf81a70689f4c79362d44c78d9e2e563611fc57e524df3"} err="failed to get container status \"b409b192f53cda255baf81a70689f4c79362d44c78d9e2e563611fc57e524df3\": rpc error: code = NotFound desc = could not find container \"b409b192f53cda255baf81a70689f4c79362d44c78d9e2e563611fc57e524df3\": container with ID starting with b409b192f53cda255baf81a70689f4c79362d44c78d9e2e563611fc57e524df3 not found: ID does not exist" Nov 03 03:12:28.978470 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:28.978416 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns"] Nov 03 03:12:29.028765 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:29.028736 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3a4683-67f6fc55c9-g8sns"] Nov 03 03:12:29.095106 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:29.095076 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8"] Nov 03 03:12:29.097806 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:29.097786 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3a4683-5ffc57ff49-6hpg8"] Nov 03 03:12:29.588216 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:29.588169 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:30.415970 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:30.415935 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt"] Nov 03 03:12:30.416510 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:30.416258 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" podUID="ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" containerName="istio-proxy" containerID="cri-o://83432dcd187bdd3545a0d148656f90056b81f9b119ea6e62f75f0647bd862237" gracePeriod=30 Nov 03 03:12:30.443627 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:30.443601 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d683621e-ee05-4efe-9d28-49d1f085d135" path="/var/lib/kubelet/pods/d683621e-ee05-4efe-9d28-49d1f085d135/volumes" Nov 03 03:12:30.443980 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:30.443967 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da6e7a0f-9fd7-4115-9c62-caed63f180de" path="/var/lib/kubelet/pods/da6e7a0f-9fd7-4115-9c62-caed63f180de/volumes" Nov 03 03:12:30.656759 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:30.656722 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:31.630275 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:31.630242 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:32.084882 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:32.084843 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-dfzjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:12:32.085060 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:32.084903 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" podUID="ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:12:33.854806 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:33.854773 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:34.029815 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:34.029731 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:34.486217 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:34.486156 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv" podUID="ed76fc49-8f0a-4a64-a273-01cb7a2d612b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:12:35.473951 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.473916 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:35.682494 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.682468 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 03:12:35.791465 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.791381 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-credential-socket\") pod \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " Nov 03 03:12:35.791465 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.791425 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-kube-api-access-v58dh\" (UniqueName: \"kubernetes.io/projected/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-kube-api-access-v58dh\") pod \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " Nov 03 03:12:35.791465 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.791452 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-workload-certs\") pod \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " Nov 03 03:12:35.791738 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.791488 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-podinfo\") pod \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " Nov 03 03:12:35.791738 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.791565 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-token\" (UniqueName: \"kubernetes.io/projected/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-token\") pod \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " Nov 03 03:12:35.791738 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.791584 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-data\") pod \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " Nov 03 03:12:35.791738 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.791611 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istiod-ca-cert\") pod \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " Nov 03 03:12:35.791738 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.791674 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-workload-socket\") pod \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " Nov 03 03:12:35.791738 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.791692 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-credential-socket" pod "ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" (UID: "ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:35.791738 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.791704 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-envoy\") pod \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\" (UID: \"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d\") " Nov 03 03:12:35.792266 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.792171 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:35.792412 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.792391 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-workload-socket" pod "ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" (UID: "ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:35.792490 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.792411 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istiod-ca-cert" pod "ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" (UID: "ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:12:35.792722 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.792653 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-workload-certs" pod "ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" (UID: "ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:35.792839 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.792732 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-data" pod "ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" (UID: "ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:35.794272 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.794235 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-token" pod "ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" (UID: "ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:35.794871 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.794844 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-envoy" pod "ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" (UID: "ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:35.795001 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.794977 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-kube-api-access-v58dh" pod "ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" (UID: "ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d"). InnerVolumeSpecName "kube-api-access-v58dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:35.795113 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.795092 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-podinfo" pod "ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" (UID: "ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:12:35.859626 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.858694 5463 generic.go:356] "Generic (PLEG): container finished" podID="ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" containerID="83432dcd187bdd3545a0d148656f90056b81f9b119ea6e62f75f0647bd862237" exitCode=0 Nov 03 03:12:35.859626 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.858802 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" event={"ID":"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d","Type":"ContainerDied","Data":"83432dcd187bdd3545a0d148656f90056b81f9b119ea6e62f75f0647bd862237"} Nov 03 03:12:35.859626 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.858888 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" event={"ID":"ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d","Type":"ContainerDied","Data":"1b45bdad89f5a19ba10c4928c73654ce2eecbc2ab5408bc931f02a739c2cd64f"} Nov 03 03:12:35.859626 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.858917 5463 scope.go:122] "RemoveContainer" containerID="83432dcd187bdd3545a0d148656f90056b81f9b119ea6e62f75f0647bd862237" Nov 03 03:12:35.859626 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.859362 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt" Nov 03 03:12:35.872224 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.872172 5463 scope.go:122] "RemoveContainer" containerID="83432dcd187bdd3545a0d148656f90056b81f9b119ea6e62f75f0647bd862237" Nov 03 03:12:35.872613 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:12:35.872588 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83432dcd187bdd3545a0d148656f90056b81f9b119ea6e62f75f0647bd862237\": container with ID starting with 83432dcd187bdd3545a0d148656f90056b81f9b119ea6e62f75f0647bd862237 not found: ID does not exist" containerID="83432dcd187bdd3545a0d148656f90056b81f9b119ea6e62f75f0647bd862237" Nov 03 03:12:35.872683 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.872625 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83432dcd187bdd3545a0d148656f90056b81f9b119ea6e62f75f0647bd862237"} err="failed to get container status \"83432dcd187bdd3545a0d148656f90056b81f9b119ea6e62f75f0647bd862237\": rpc error: code = NotFound desc = could not find container \"83432dcd187bdd3545a0d148656f90056b81f9b119ea6e62f75f0647bd862237\": container with ID starting with 83432dcd187bdd3545a0d148656f90056b81f9b119ea6e62f75f0647bd862237 not found: ID does not exist" Nov 03 03:12:35.892945 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.892913 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:35.892945 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.892939 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:35.893095 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.892954 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:35.893095 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.892967 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:35.893095 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.892979 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:35.893095 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.892993 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v58dh\" (UniqueName: \"kubernetes.io/projected/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-kube-api-access-v58dh\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:35.893095 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.893008 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:35.893095 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.893021 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:35.941767 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.941735 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs"] Nov 03 03:12:35.942012 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.941981 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" podUID="a26968f3-5698-4639-aa81-bedc87536bd6" containerName="istio-proxy" containerID="cri-o://ed88cf210f0fde048e7a0868854a4e698095edad0dc9e498075a5e8953a9cb48" gracePeriod=30 Nov 03 03:12:35.998059 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:35.998030 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:36.032494 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:36.032465 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt"] Nov 03 03:12:36.037511 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:36.037487 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-dfzjt"] Nov 03 03:12:36.442929 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:36.442900 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" path="/var/lib/kubelet/pods/ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d/volumes" Nov 03 03:12:36.454060 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:36.454036 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:36.604003 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:36.603971 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:36.629372 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:36.629339 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:12:36.629543 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:36.629408 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" podUID="a26968f3-5698-4639-aa81-bedc87536bd6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:12:40.362012 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:40.361979 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv"] Nov 03 03:12:40.394993 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:40.394962 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:41.190287 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.190262 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 03:12:41.259611 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.259545 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a26968f3-5698-4639-aa81-bedc87536bd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26968f3-5698-4639-aa81-bedc87536bd6-istiod-ca-cert\") pod \"a26968f3-5698-4639-aa81-bedc87536bd6\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " Nov 03 03:12:41.259611 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.259597 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a26968f3-5698-4639-aa81-bedc87536bd6-kube-api-access-dtdzk\" (UniqueName: \"kubernetes.io/projected/a26968f3-5698-4639-aa81-bedc87536bd6-kube-api-access-dtdzk\") pod \"a26968f3-5698-4639-aa81-bedc87536bd6\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " Nov 03 03:12:41.259845 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.259654 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a26968f3-5698-4639-aa81-bedc87536bd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26968f3-5698-4639-aa81-bedc87536bd6-istio-podinfo\") pod \"a26968f3-5698-4639-aa81-bedc87536bd6\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " Nov 03 03:12:41.259845 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.259676 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-credential-socket\") pod \"a26968f3-5698-4639-aa81-bedc87536bd6\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " Nov 03 03:12:41.259845 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.259712 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-istio-data\") pod \"a26968f3-5698-4639-aa81-bedc87536bd6\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " Nov 03 03:12:41.259845 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.259777 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a26968f3-5698-4639-aa81-bedc87536bd6-istio-token\" (UniqueName: \"kubernetes.io/projected/a26968f3-5698-4639-aa81-bedc87536bd6-istio-token\") pod \"a26968f3-5698-4639-aa81-bedc87536bd6\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " Nov 03 03:12:41.259845 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.259808 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-istio-envoy\") pod \"a26968f3-5698-4639-aa81-bedc87536bd6\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " Nov 03 03:12:41.260129 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.259905 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-workload-certs\") pod \"a26968f3-5698-4639-aa81-bedc87536bd6\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " Nov 03 03:12:41.260129 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.259947 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-workload-socket\") pod \"a26968f3-5698-4639-aa81-bedc87536bd6\" (UID: \"a26968f3-5698-4639-aa81-bedc87536bd6\") " Nov 03 03:12:41.260129 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.259965 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a26968f3-5698-4639-aa81-bedc87536bd6-istiod-ca-cert" pod "a26968f3-5698-4639-aa81-bedc87536bd6" (UID: "a26968f3-5698-4639-aa81-bedc87536bd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:12:41.260355 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.260156 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-istio-data" pod "a26968f3-5698-4639-aa81-bedc87536bd6" (UID: "a26968f3-5698-4639-aa81-bedc87536bd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:41.260414 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.260388 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:41.260414 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.260409 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26968f3-5698-4639-aa81-bedc87536bd6-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:41.260536 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.260412 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-credential-socket" pod "a26968f3-5698-4639-aa81-bedc87536bd6" (UID: "a26968f3-5698-4639-aa81-bedc87536bd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:41.260617 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.260596 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-workload-certs" pod "a26968f3-5698-4639-aa81-bedc87536bd6" (UID: "a26968f3-5698-4639-aa81-bedc87536bd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:41.260703 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.260648 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-workload-socket" pod "a26968f3-5698-4639-aa81-bedc87536bd6" (UID: "a26968f3-5698-4639-aa81-bedc87536bd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:41.262456 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.262429 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-istio-envoy" pod "a26968f3-5698-4639-aa81-bedc87536bd6" (UID: "a26968f3-5698-4639-aa81-bedc87536bd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:41.262821 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.262794 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26968f3-5698-4639-aa81-bedc87536bd6-istio-token" pod "a26968f3-5698-4639-aa81-bedc87536bd6" (UID: "a26968f3-5698-4639-aa81-bedc87536bd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:41.262943 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.262839 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a26968f3-5698-4639-aa81-bedc87536bd6-istio-podinfo" pod "a26968f3-5698-4639-aa81-bedc87536bd6" (UID: "a26968f3-5698-4639-aa81-bedc87536bd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:12:41.262943 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.262923 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26968f3-5698-4639-aa81-bedc87536bd6-kube-api-access-dtdzk" pod "a26968f3-5698-4639-aa81-bedc87536bd6" (UID: "a26968f3-5698-4639-aa81-bedc87536bd6"). InnerVolumeSpecName "kube-api-access-dtdzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:41.361329 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.361302 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a26968f3-5698-4639-aa81-bedc87536bd6-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:41.361329 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.361329 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:41.361557 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.361344 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:41.361557 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.361359 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:41.361557 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.361372 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtdzk\" (UniqueName: \"kubernetes.io/projected/a26968f3-5698-4639-aa81-bedc87536bd6-kube-api-access-dtdzk\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:41.361557 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.361386 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26968f3-5698-4639-aa81-bedc87536bd6-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:41.361557 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.361401 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26968f3-5698-4639-aa81-bedc87536bd6-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:41.413000 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.412971 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw"] Nov 03 03:12:41.415498 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.415472 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p"] Nov 03 03:12:41.525715 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.525641 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:41.682950 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.682916 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:41.900879 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.900846 5463 generic.go:356] "Generic (PLEG): container finished" podID="a26968f3-5698-4639-aa81-bedc87536bd6" containerID="ed88cf210f0fde048e7a0868854a4e698095edad0dc9e498075a5e8953a9cb48" exitCode=0 Nov 03 03:12:41.901068 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.900901 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" Nov 03 03:12:41.901068 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.900927 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" event={"ID":"a26968f3-5698-4639-aa81-bedc87536bd6","Type":"ContainerDied","Data":"ed88cf210f0fde048e7a0868854a4e698095edad0dc9e498075a5e8953a9cb48"} Nov 03 03:12:41.901068 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.900984 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs" event={"ID":"a26968f3-5698-4639-aa81-bedc87536bd6","Type":"ContainerDied","Data":"6b6af5741995b4ffd6938ddc3ff4bd8fff80af13fc17e95825896186fc192dff"} Nov 03 03:12:41.901068 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.901014 5463 scope.go:122] "RemoveContainer" containerID="ed88cf210f0fde048e7a0868854a4e698095edad0dc9e498075a5e8953a9cb48" Nov 03 03:12:41.911705 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.911689 5463 scope.go:122] "RemoveContainer" containerID="31489e3b391bc6c54f6718e36e57579f1d901c82194680c6692bde75fdeaeeff" Nov 03 03:12:41.920705 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.920689 5463 scope.go:122] "RemoveContainer" containerID="ed88cf210f0fde048e7a0868854a4e698095edad0dc9e498075a5e8953a9cb48" Nov 03 03:12:41.920997 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:12:41.920972 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed88cf210f0fde048e7a0868854a4e698095edad0dc9e498075a5e8953a9cb48\": container with ID starting with ed88cf210f0fde048e7a0868854a4e698095edad0dc9e498075a5e8953a9cb48 not found: ID does not exist" containerID="ed88cf210f0fde048e7a0868854a4e698095edad0dc9e498075a5e8953a9cb48" Nov 03 03:12:41.921109 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.921002 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed88cf210f0fde048e7a0868854a4e698095edad0dc9e498075a5e8953a9cb48"} err="failed to get container status \"ed88cf210f0fde048e7a0868854a4e698095edad0dc9e498075a5e8953a9cb48\": rpc error: code = NotFound desc = could not find container \"ed88cf210f0fde048e7a0868854a4e698095edad0dc9e498075a5e8953a9cb48\": container with ID starting with ed88cf210f0fde048e7a0868854a4e698095edad0dc9e498075a5e8953a9cb48 not found: ID does not exist" Nov 03 03:12:41.921109 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.921022 5463 scope.go:122] "RemoveContainer" containerID="31489e3b391bc6c54f6718e36e57579f1d901c82194680c6692bde75fdeaeeff" Nov 03 03:12:41.921345 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:12:41.921328 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31489e3b391bc6c54f6718e36e57579f1d901c82194680c6692bde75fdeaeeff\": container with ID starting with 31489e3b391bc6c54f6718e36e57579f1d901c82194680c6692bde75fdeaeeff not found: ID does not exist" containerID="31489e3b391bc6c54f6718e36e57579f1d901c82194680c6692bde75fdeaeeff" Nov 03 03:12:41.921392 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:41.921354 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31489e3b391bc6c54f6718e36e57579f1d901c82194680c6692bde75fdeaeeff"} err="failed to get container status \"31489e3b391bc6c54f6718e36e57579f1d901c82194680c6692bde75fdeaeeff\": rpc error: code = NotFound desc = could not find container \"31489e3b391bc6c54f6718e36e57579f1d901c82194680c6692bde75fdeaeeff\": container with ID starting with 31489e3b391bc6c54f6718e36e57579f1d901c82194680c6692bde75fdeaeeff not found: ID does not exist" Nov 03 03:12:42.066016 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:42.065984 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs"] Nov 03 03:12:42.069361 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:42.069337 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ffba43-openshift-default-7d99dcb89-cktrs"] Nov 03 03:12:42.442629 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:42.442596 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a26968f3-5698-4639-aa81-bedc87536bd6" path="/var/lib/kubelet/pods/a26968f3-5698-4639-aa81-bedc87536bd6/volumes" Nov 03 03:12:43.007845 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:12:43.007813 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:12:43.421608 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:43.421568 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:43.897029 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:43.896997 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:44.486207 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:44.486156 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv" podUID="ed76fc49-8f0a-4a64-a273-01cb7a2d612b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:12:46.265245 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:46.265207 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:46.436353 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:46.436321 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:48.280161 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:48.280126 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:48.441852 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:48.441824 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:48.911094 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:48.911059 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:48.920234 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:48.920215 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv" Nov 03 03:12:48.946947 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:48.946913 5463 generic.go:356] "Generic (PLEG): container finished" podID="ed76fc49-8f0a-4a64-a273-01cb7a2d612b" containerID="936a662c6ae831a69413aaeedfa9309906acc3a4bd8b88fefc2a6dbe6436d34e" exitCode=0 Nov 03 03:12:48.947102 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:48.946962 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv" Nov 03 03:12:48.947102 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:48.946963 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv" event={"ID":"ed76fc49-8f0a-4a64-a273-01cb7a2d612b","Type":"ContainerDied","Data":"936a662c6ae831a69413aaeedfa9309906acc3a4bd8b88fefc2a6dbe6436d34e"} Nov 03 03:12:48.947102 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:48.947034 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv" event={"ID":"ed76fc49-8f0a-4a64-a273-01cb7a2d612b","Type":"ContainerDied","Data":"a9d0618bcfb9714974310c49214618caf7f7009273dc5e5e634d9fc5dd081e78"} Nov 03 03:12:48.947102 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:48.947064 5463 scope.go:122] "RemoveContainer" containerID="936a662c6ae831a69413aaeedfa9309906acc3a4bd8b88fefc2a6dbe6436d34e" Nov 03 03:12:48.958489 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:48.958474 5463 scope.go:122] "RemoveContainer" containerID="936a662c6ae831a69413aaeedfa9309906acc3a4bd8b88fefc2a6dbe6436d34e" Nov 03 03:12:48.958753 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:12:48.958733 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"936a662c6ae831a69413aaeedfa9309906acc3a4bd8b88fefc2a6dbe6436d34e\": container with ID starting with 936a662c6ae831a69413aaeedfa9309906acc3a4bd8b88fefc2a6dbe6436d34e not found: ID does not exist" containerID="936a662c6ae831a69413aaeedfa9309906acc3a4bd8b88fefc2a6dbe6436d34e" Nov 03 03:12:48.958841 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:48.958760 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"936a662c6ae831a69413aaeedfa9309906acc3a4bd8b88fefc2a6dbe6436d34e"} err="failed to get container status \"936a662c6ae831a69413aaeedfa9309906acc3a4bd8b88fefc2a6dbe6436d34e\": rpc error: code = NotFound desc = could not find container \"936a662c6ae831a69413aaeedfa9309906acc3a4bd8b88fefc2a6dbe6436d34e\": container with ID starting with 936a662c6ae831a69413aaeedfa9309906acc3a4bd8b88fefc2a6dbe6436d34e not found: ID does not exist" Nov 03 03:12:49.045803 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:49.045768 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed76fc49-8f0a-4a64-a273-01cb7a2d612b-kube-api-access-dnsnx\" (UniqueName: \"kubernetes.io/projected/ed76fc49-8f0a-4a64-a273-01cb7a2d612b-kube-api-access-dnsnx\") pod \"ed76fc49-8f0a-4a64-a273-01cb7a2d612b\" (UID: \"ed76fc49-8f0a-4a64-a273-01cb7a2d612b\") " Nov 03 03:12:49.046002 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:49.045902 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed76fc49-8f0a-4a64-a273-01cb7a2d612b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed76fc49-8f0a-4a64-a273-01cb7a2d612b-plugins-config-volume\") pod \"ed76fc49-8f0a-4a64-a273-01cb7a2d612b\" (UID: \"ed76fc49-8f0a-4a64-a273-01cb7a2d612b\") " Nov 03 03:12:49.046286 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:49.046264 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed76fc49-8f0a-4a64-a273-01cb7a2d612b-plugins-config-volume" pod "ed76fc49-8f0a-4a64-a273-01cb7a2d612b" (UID: "ed76fc49-8f0a-4a64-a273-01cb7a2d612b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:12:49.046404 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:49.046386 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed76fc49-8f0a-4a64-a273-01cb7a2d612b-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:49.048608 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:49.048572 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed76fc49-8f0a-4a64-a273-01cb7a2d612b-kube-api-access-dnsnx" pod "ed76fc49-8f0a-4a64-a273-01cb7a2d612b" (UID: "ed76fc49-8f0a-4a64-a273-01cb7a2d612b"). InnerVolumeSpecName "kube-api-access-dnsnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:49.147826 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:49.147791 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnsnx\" (UniqueName: \"kubernetes.io/projected/ed76fc49-8f0a-4a64-a273-01cb7a2d612b-kube-api-access-dnsnx\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:49.374974 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:49.374935 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv"] Nov 03 03:12:49.382421 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:49.382385 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-78b5c5-797d54b9ff-ldvhv"] Nov 03 03:12:50.445011 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:50.444977 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed76fc49-8f0a-4a64-a273-01cb7a2d612b" path="/var/lib/kubelet/pods/ed76fc49-8f0a-4a64-a273-01cb7a2d612b/volumes" Nov 03 03:12:50.509304 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:50.509266 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:52.130136 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:52.130108 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:52.274433 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:52.274402 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:53.007897 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:53.007863 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc"] Nov 03 03:12:53.114597 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:53.114566 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst"] Nov 03 03:12:53.390769 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:53.390685 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:53.546163 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:53.546132 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:54.050203 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:54.050164 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:54.206624 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:54.206590 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:55.382001 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.381924 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" podUID="53646ed7-2ee1-4a28-9242-033c5be233e4" containerName="main" containerID="cri-o://1fc63374ac54e365dc96a2f0448097632f22dc9a6be8ca640e70041e31ea0412" gracePeriod=45 Nov 03 03:12:55.566671 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.566603 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:55.636344 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.636287 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:12:55.713053 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.713012 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-home\" (UniqueName: \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-home\") pod \"53646ed7-2ee1-4a28-9242-033c5be233e4\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " Nov 03 03:12:55.713053 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.713051 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53646ed7-2ee1-4a28-9242-033c5be233e4-kube-api-access-d78fg\" (UniqueName: \"kubernetes.io/projected/53646ed7-2ee1-4a28-9242-033c5be233e4-kube-api-access-d78fg\") pod \"53646ed7-2ee1-4a28-9242-033c5be233e4\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " Nov 03 03:12:55.715436 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.713182 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53646ed7-2ee1-4a28-9242-033c5be233e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/53646ed7-2ee1-4a28-9242-033c5be233e4-tls-certs\") pod \"53646ed7-2ee1-4a28-9242-033c5be233e4\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " Nov 03 03:12:55.715436 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.713298 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-tmp-dir\") pod \"53646ed7-2ee1-4a28-9242-033c5be233e4\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " Nov 03 03:12:55.715436 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.713358 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-home" pod "53646ed7-2ee1-4a28-9242-033c5be233e4" (UID: "53646ed7-2ee1-4a28-9242-033c5be233e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:55.715436 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.713434 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-dshm\") pod \"53646ed7-2ee1-4a28-9242-033c5be233e4\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " Nov 03 03:12:55.715436 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.713472 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-model-cache\") pod \"53646ed7-2ee1-4a28-9242-033c5be233e4\" (UID: \"53646ed7-2ee1-4a28-9242-033c5be233e4\") " Nov 03 03:12:55.715436 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.713605 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-tmp-dir" pod "53646ed7-2ee1-4a28-9242-033c5be233e4" (UID: "53646ed7-2ee1-4a28-9242-033c5be233e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:55.715436 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.713814 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-model-cache" pod "53646ed7-2ee1-4a28-9242-033c5be233e4" (UID: "53646ed7-2ee1-4a28-9242-033c5be233e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:55.715436 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.713894 5463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-tmp-dir\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:55.715436 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.713915 5463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-model-cache\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:55.715436 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.713924 5463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-home\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:55.715865 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.715595 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-dshm" pod "53646ed7-2ee1-4a28-9242-033c5be233e4" (UID: "53646ed7-2ee1-4a28-9242-033c5be233e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:55.715865 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.715760 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53646ed7-2ee1-4a28-9242-033c5be233e4-kube-api-access-d78fg" pod "53646ed7-2ee1-4a28-9242-033c5be233e4" (UID: "53646ed7-2ee1-4a28-9242-033c5be233e4"). InnerVolumeSpecName "kube-api-access-d78fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:55.715865 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.715824 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53646ed7-2ee1-4a28-9242-033c5be233e4-tls-certs" pod "53646ed7-2ee1-4a28-9242-033c5be233e4" (UID: "53646ed7-2ee1-4a28-9242-033c5be233e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:12:55.814741 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.814707 5463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53646ed7-2ee1-4a28-9242-033c5be233e4-dshm\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:55.814741 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.814743 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d78fg\" (UniqueName: \"kubernetes.io/projected/53646ed7-2ee1-4a28-9242-033c5be233e4-kube-api-access-d78fg\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:55.814949 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.814760 5463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53646ed7-2ee1-4a28-9242-033c5be233e4-tls-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:55.995809 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.995782 5463 generic.go:356] "Generic (PLEG): container finished" podID="53646ed7-2ee1-4a28-9242-033c5be233e4" containerID="1fc63374ac54e365dc96a2f0448097632f22dc9a6be8ca640e70041e31ea0412" exitCode=0 Nov 03 03:12:55.996011 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.995839 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" Nov 03 03:12:55.996011 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.995871 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" event={"ID":"53646ed7-2ee1-4a28-9242-033c5be233e4","Type":"ContainerDied","Data":"1fc63374ac54e365dc96a2f0448097632f22dc9a6be8ca640e70041e31ea0412"} Nov 03 03:12:55.996011 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.995921 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv" event={"ID":"53646ed7-2ee1-4a28-9242-033c5be233e4","Type":"ContainerDied","Data":"ed37cbe04148c19a97c6ad969d215b0841cb6913255b0a5c46cd537265df4302"} Nov 03 03:12:55.996011 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:55.995946 5463 scope.go:122] "RemoveContainer" containerID="1fc63374ac54e365dc96a2f0448097632f22dc9a6be8ca640e70041e31ea0412" Nov 03 03:12:56.013079 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.013059 5463 scope.go:122] "RemoveContainer" containerID="1fc63374ac54e365dc96a2f0448097632f22dc9a6be8ca640e70041e31ea0412" Nov 03 03:12:56.013381 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:12:56.013362 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fc63374ac54e365dc96a2f0448097632f22dc9a6be8ca640e70041e31ea0412\": container with ID starting with 1fc63374ac54e365dc96a2f0448097632f22dc9a6be8ca640e70041e31ea0412 not found: ID does not exist" containerID="1fc63374ac54e365dc96a2f0448097632f22dc9a6be8ca640e70041e31ea0412" Nov 03 03:12:56.013455 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.013392 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fc63374ac54e365dc96a2f0448097632f22dc9a6be8ca640e70041e31ea0412"} err="failed to get container status \"1fc63374ac54e365dc96a2f0448097632f22dc9a6be8ca640e70041e31ea0412\": rpc error: code = NotFound desc = could not find container \"1fc63374ac54e365dc96a2f0448097632f22dc9a6be8ca640e70041e31ea0412\": container with ID starting with 1fc63374ac54e365dc96a2f0448097632f22dc9a6be8ca640e70041e31ea0412 not found: ID does not exist" Nov 03 03:12:56.102149 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.102116 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w"] Nov 03 03:12:56.102428 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.102385 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" podUID="8f8b0d5f-94b6-439a-bbb0-35fb828131dc" containerName="istio-proxy" containerID="cri-o://f03db564fe7b11373e7598a551d7d2fc033384c465a12387536d243ba12b8082" gracePeriod=30 Nov 03 03:12:56.143483 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.143451 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:56.166094 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.166059 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv"] Nov 03 03:12:56.170849 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.170820 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f735f4/sub-test-model-f735f4-kserve-85f9f7b767-5xkgv"] Nov 03 03:12:56.433527 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.433487 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" podUID="1ba01dba-9d25-480d-b5d1-f51295ed81b7" containerName="main" containerID="cri-o://39599de2064b6ff9abcba65459238475edaf70d43135de4189cdb08da3d3d65e" gracePeriod=45 Nov 03 03:12:56.434743 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.434716 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" containerID="cri-o://dc8e9c83ba6d266504f57cf11a83c015bc55e2fa683ad7727a3651368455c96b" gracePeriod=45 Nov 03 03:12:56.442953 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.442927 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53646ed7-2ee1-4a28-9242-033c5be233e4" path="/var/lib/kubelet/pods/53646ed7-2ee1-4a28-9242-033c5be233e4/volumes" Nov 03 03:12:56.624815 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.624766 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:56.699896 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.699862 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 03:12:56.715519 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.715497 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 03:12:56.823765 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.823737 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-model-cache\") pod \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " Nov 03 03:12:56.823969 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.823797 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-dshm\") pod \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " Nov 03 03:12:56.823969 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.823820 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ba01dba-9d25-480d-b5d1-f51295ed81b7-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ba01dba-9d25-480d-b5d1-f51295ed81b7-tls-certs\") pod \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " Nov 03 03:12:56.823969 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.823866 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-home\" (UniqueName: \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-home\") pod \"230d658f-a760-4d30-92f4-b021eda43859\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " Nov 03 03:12:56.823969 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.823892 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/230d658f-a760-4d30-92f4-b021eda43859-tls-certs\" (UniqueName: \"kubernetes.io/secret/230d658f-a760-4d30-92f4-b021eda43859-tls-certs\") pod \"230d658f-a760-4d30-92f4-b021eda43859\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " Nov 03 03:12:56.823969 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.823938 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-home\" (UniqueName: \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-home\") pod \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " Nov 03 03:12:56.824299 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.823984 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-tmp-dir\") pod \"230d658f-a760-4d30-92f4-b021eda43859\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " Nov 03 03:12:56.824299 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.824040 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-tmp-dir\") pod \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " Nov 03 03:12:56.824299 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.824049 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-model-cache" pod "1ba01dba-9d25-480d-b5d1-f51295ed81b7" (UID: "1ba01dba-9d25-480d-b5d1-f51295ed81b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:56.824299 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.824087 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-home" pod "230d658f-a760-4d30-92f4-b021eda43859" (UID: "230d658f-a760-4d30-92f4-b021eda43859"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:56.824299 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.824107 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba01dba-9d25-480d-b5d1-f51295ed81b7-kube-api-access-sqbnk\" (UniqueName: \"kubernetes.io/projected/1ba01dba-9d25-480d-b5d1-f51295ed81b7-kube-api-access-sqbnk\") pod \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\" (UID: \"1ba01dba-9d25-480d-b5d1-f51295ed81b7\") " Nov 03 03:12:56.824299 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.824149 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-model-cache\") pod \"230d658f-a760-4d30-92f4-b021eda43859\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " Nov 03 03:12:56.824299 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.824237 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230d658f-a760-4d30-92f4-b021eda43859-kube-api-access-cm49p\" (UniqueName: \"kubernetes.io/projected/230d658f-a760-4d30-92f4-b021eda43859-kube-api-access-cm49p\") pod \"230d658f-a760-4d30-92f4-b021eda43859\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " Nov 03 03:12:56.824299 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.824277 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-dshm\" (UniqueName: \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-dshm\") pod \"230d658f-a760-4d30-92f4-b021eda43859\" (UID: \"230d658f-a760-4d30-92f4-b021eda43859\") " Nov 03 03:12:56.824764 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.824339 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-home" pod "1ba01dba-9d25-480d-b5d1-f51295ed81b7" (UID: "1ba01dba-9d25-480d-b5d1-f51295ed81b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:56.824764 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.824619 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-tmp-dir" pod "230d658f-a760-4d30-92f4-b021eda43859" (UID: "230d658f-a760-4d30-92f4-b021eda43859"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:56.824764 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.824669 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-model-cache" pod "230d658f-a760-4d30-92f4-b021eda43859" (UID: "230d658f-a760-4d30-92f4-b021eda43859"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:56.824764 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.824685 5463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-home\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.824764 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.824704 5463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-home\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.824764 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.824696 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-tmp-dir" pod "1ba01dba-9d25-480d-b5d1-f51295ed81b7" (UID: "1ba01dba-9d25-480d-b5d1-f51295ed81b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:56.824764 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.824719 5463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-model-cache\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.826965 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.826943 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ba01dba-9d25-480d-b5d1-f51295ed81b7-tls-certs" pod "1ba01dba-9d25-480d-b5d1-f51295ed81b7" (UID: "1ba01dba-9d25-480d-b5d1-f51295ed81b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:12:56.827105 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.826977 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba01dba-9d25-480d-b5d1-f51295ed81b7-kube-api-access-sqbnk" pod "1ba01dba-9d25-480d-b5d1-f51295ed81b7" (UID: "1ba01dba-9d25-480d-b5d1-f51295ed81b7"). InnerVolumeSpecName "kube-api-access-sqbnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:56.827364 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.827338 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/230d658f-a760-4d30-92f4-b021eda43859-tls-certs" pod "230d658f-a760-4d30-92f4-b021eda43859" (UID: "230d658f-a760-4d30-92f4-b021eda43859"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:12:56.827473 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.827391 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-dshm" pod "230d658f-a760-4d30-92f4-b021eda43859" (UID: "230d658f-a760-4d30-92f4-b021eda43859"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:56.827902 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.827883 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230d658f-a760-4d30-92f4-b021eda43859-kube-api-access-cm49p" pod "230d658f-a760-4d30-92f4-b021eda43859" (UID: "230d658f-a760-4d30-92f4-b021eda43859"). InnerVolumeSpecName "kube-api-access-cm49p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:56.827968 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.827924 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-dshm" pod "1ba01dba-9d25-480d-b5d1-f51295ed81b7" (UID: "1ba01dba-9d25-480d-b5d1-f51295ed81b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:12:56.926232 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.926165 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqbnk\" (UniqueName: \"kubernetes.io/projected/1ba01dba-9d25-480d-b5d1-f51295ed81b7-kube-api-access-sqbnk\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.926232 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.926216 5463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-model-cache\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.926232 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.926232 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm49p\" (UniqueName: \"kubernetes.io/projected/230d658f-a760-4d30-92f4-b021eda43859-kube-api-access-cm49p\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.926511 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.926247 5463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-dshm\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.926511 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.926262 5463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-dshm\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.926511 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.926273 5463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ba01dba-9d25-480d-b5d1-f51295ed81b7-tls-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.926511 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.926285 5463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230d658f-a760-4d30-92f4-b021eda43859-tls-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.926511 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.926297 5463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230d658f-a760-4d30-92f4-b021eda43859-tmp-dir\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:56.926511 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:56.926307 5463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ba01dba-9d25-480d-b5d1-f51295ed81b7-tmp-dir\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:57.002997 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:57.002922 5463 generic.go:356] "Generic (PLEG): container finished" podID="1ba01dba-9d25-480d-b5d1-f51295ed81b7" containerID="39599de2064b6ff9abcba65459238475edaf70d43135de4189cdb08da3d3d65e" exitCode=0 Nov 03 03:12:57.002997 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:57.002969 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" Nov 03 03:12:57.003184 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:57.003003 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" event={"ID":"1ba01dba-9d25-480d-b5d1-f51295ed81b7","Type":"ContainerDied","Data":"39599de2064b6ff9abcba65459238475edaf70d43135de4189cdb08da3d3d65e"} Nov 03 03:12:57.003184 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:57.003046 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw" event={"ID":"1ba01dba-9d25-480d-b5d1-f51295ed81b7","Type":"ContainerDied","Data":"97693b516a6dff9535d9c52b9e2af9b6d9b62edb6342dd1ce5f368449cc7ca1b"} Nov 03 03:12:57.003184 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:57.003063 5463 scope.go:122] "RemoveContainer" containerID="39599de2064b6ff9abcba65459238475edaf70d43135de4189cdb08da3d3d65e" Nov 03 03:12:57.004415 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:57.004392 5463 generic.go:356] "Generic (PLEG): container finished" podID="230d658f-a760-4d30-92f4-b021eda43859" containerID="dc8e9c83ba6d266504f57cf11a83c015bc55e2fa683ad7727a3651368455c96b" exitCode=0 Nov 03 03:12:57.004531 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:57.004442 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" Nov 03 03:12:57.004531 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:57.004484 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" event={"ID":"230d658f-a760-4d30-92f4-b021eda43859","Type":"ContainerDied","Data":"dc8e9c83ba6d266504f57cf11a83c015bc55e2fa683ad7727a3651368455c96b"} Nov 03 03:12:57.004531 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:57.004521 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p" event={"ID":"230d658f-a760-4d30-92f4-b021eda43859","Type":"ContainerDied","Data":"cd93d181cfa62de2babf0515eda3c9a167e628501752dfffdc3b7093bc670abb"} Nov 03 03:12:57.014923 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:57.014899 5463 scope.go:122] "RemoveContainer" containerID="39599de2064b6ff9abcba65459238475edaf70d43135de4189cdb08da3d3d65e" Nov 03 03:12:57.015287 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:12:57.015261 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39599de2064b6ff9abcba65459238475edaf70d43135de4189cdb08da3d3d65e\": container with ID starting with 39599de2064b6ff9abcba65459238475edaf70d43135de4189cdb08da3d3d65e not found: ID does not exist" containerID="39599de2064b6ff9abcba65459238475edaf70d43135de4189cdb08da3d3d65e" Nov 03 03:12:57.015388 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:57.015300 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39599de2064b6ff9abcba65459238475edaf70d43135de4189cdb08da3d3d65e"} err="failed to get container status \"39599de2064b6ff9abcba65459238475edaf70d43135de4189cdb08da3d3d65e\": rpc error: code = NotFound desc = could not find container \"39599de2064b6ff9abcba65459238475edaf70d43135de4189cdb08da3d3d65e\": container with ID starting with 39599de2064b6ff9abcba65459238475edaf70d43135de4189cdb08da3d3d65e not found: ID does not exist" Nov 03 03:12:57.015388 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:57.015322 5463 scope.go:122] "RemoveContainer" containerID="dc8e9c83ba6d266504f57cf11a83c015bc55e2fa683ad7727a3651368455c96b" Nov 03 03:12:57.025684 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:57.025663 5463 scope.go:122] "RemoveContainer" containerID="dc8e9c83ba6d266504f57cf11a83c015bc55e2fa683ad7727a3651368455c96b" Nov 03 03:12:57.025966 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:12:57.025940 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc8e9c83ba6d266504f57cf11a83c015bc55e2fa683ad7727a3651368455c96b\": container with ID starting with dc8e9c83ba6d266504f57cf11a83c015bc55e2fa683ad7727a3651368455c96b not found: ID does not exist" containerID="dc8e9c83ba6d266504f57cf11a83c015bc55e2fa683ad7727a3651368455c96b" Nov 03 03:12:57.026044 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:57.025973 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc8e9c83ba6d266504f57cf11a83c015bc55e2fa683ad7727a3651368455c96b"} err="failed to get container status \"dc8e9c83ba6d266504f57cf11a83c015bc55e2fa683ad7727a3651368455c96b\": rpc error: code = NotFound desc = could not find container \"dc8e9c83ba6d266504f57cf11a83c015bc55e2fa683ad7727a3651368455c96b\": container with ID starting with dc8e9c83ba6d266504f57cf11a83c015bc55e2fa683ad7727a3651368455c96b not found: ID does not exist" Nov 03 03:12:57.169207 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:57.169157 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw"] Nov 03 03:12:57.173888 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:57.173858 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/distinct-2-w0-f7d9bb-kserve-7d9884cff8-7s6mw"] Nov 03 03:12:57.284448 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:57.284309 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p"] Nov 03 03:12:57.287611 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:57.287588 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f7d9bb/premium-w0-f7d9bb-kserve-7c6c9676db-hws4p"] Nov 03 03:12:57.518365 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:57.518338 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:57.815593 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:57.815557 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:58.008621 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:58.008584 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc" podUID="fcc529d1-9757-467a-bde9-c0afa4822017" containerName="payload-processing" containerID="cri-o://0fefd49e5caa378958ce5bef3a9dc5bdafe9dd5ba2e5dbbc83daa124e0e69fcd" gracePeriod=25 Nov 03 03:12:58.115408 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:58.115329 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst" podUID="e01f46d8-b6be-4f51-9291-a67a09771b4e" containerName="payload-pre-processing" containerID="cri-o://4f6d5c06c5363b6c00afe8458f67a9b558fe721d19f64abd9563c44bea0c6e5f" gracePeriod=25 Nov 03 03:12:58.442721 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:58.442688 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ba01dba-9d25-480d-b5d1-f51295ed81b7" path="/var/lib/kubelet/pods/1ba01dba-9d25-480d-b5d1-f51295ed81b7/volumes" Nov 03 03:12:58.443109 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:58.443093 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230d658f-a760-4d30-92f4-b021eda43859" path="/var/lib/kubelet/pods/230d658f-a760-4d30-92f4-b021eda43859/volumes" Nov 03 03:12:59.464823 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:59.464780 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:12:59.465305 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:12:59.464839 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" podUID="8f8b0d5f-94b6-439a-bbb0-35fb828131dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:13:01.363907 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.363878 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 03:13:01.474228 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.474174 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-workload-certs\") pod \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " Nov 03 03:13:01.474416 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.474274 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-data\") pod \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " Nov 03 03:13:01.474416 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.474336 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-podinfo\") pod \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " Nov 03 03:13:01.474416 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.474367 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-envoy\") pod \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " Nov 03 03:13:01.474416 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.474393 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istiod-ca-cert\") pod \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " Nov 03 03:13:01.474655 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.474423 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-workload-certs" pod "8f8b0d5f-94b6-439a-bbb0-35fb828131dc" (UID: "8f8b0d5f-94b6-439a-bbb0-35fb828131dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:01.474655 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.474429 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-kube-api-access-x2vgt\" (UniqueName: \"kubernetes.io/projected/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-kube-api-access-x2vgt\") pod \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " Nov 03 03:13:01.474655 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.474585 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-credential-socket\") pod \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " Nov 03 03:13:01.474836 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.474815 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-credential-socket" pod "8f8b0d5f-94b6-439a-bbb0-35fb828131dc" (UID: "8f8b0d5f-94b6-439a-bbb0-35fb828131dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:01.474903 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.474857 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istiod-ca-cert" pod "8f8b0d5f-94b6-439a-bbb0-35fb828131dc" (UID: "8f8b0d5f-94b6-439a-bbb0-35fb828131dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:13:01.474903 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.474869 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-token\" (UniqueName: \"kubernetes.io/projected/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-token\") pod \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " Nov 03 03:13:01.475018 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.474912 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-data" pod "8f8b0d5f-94b6-439a-bbb0-35fb828131dc" (UID: "8f8b0d5f-94b6-439a-bbb0-35fb828131dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:01.475018 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.474944 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-workload-socket\") pod \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\" (UID: \"8f8b0d5f-94b6-439a-bbb0-35fb828131dc\") " Nov 03 03:13:01.475324 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.475265 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-workload-socket" pod "8f8b0d5f-94b6-439a-bbb0-35fb828131dc" (UID: "8f8b0d5f-94b6-439a-bbb0-35fb828131dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:01.475484 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.475405 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:01.475484 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.475424 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:01.475484 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.475438 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:01.475484 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.475450 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:01.475484 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.475465 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:01.479382 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.479335 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-envoy" pod "8f8b0d5f-94b6-439a-bbb0-35fb828131dc" (UID: "8f8b0d5f-94b6-439a-bbb0-35fb828131dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:01.479382 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.479346 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-token" pod "8f8b0d5f-94b6-439a-bbb0-35fb828131dc" (UID: "8f8b0d5f-94b6-439a-bbb0-35fb828131dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:01.479382 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.479369 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-kube-api-access-x2vgt" pod "8f8b0d5f-94b6-439a-bbb0-35fb828131dc" (UID: "8f8b0d5f-94b6-439a-bbb0-35fb828131dc"). InnerVolumeSpecName "kube-api-access-x2vgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:01.479633 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.479460 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-podinfo" pod "8f8b0d5f-94b6-439a-bbb0-35fb828131dc" (UID: "8f8b0d5f-94b6-439a-bbb0-35fb828131dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:13:01.576710 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.576673 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:01.576710 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.576702 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:01.576710 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.576715 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:01.577036 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.576724 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2vgt\" (UniqueName: \"kubernetes.io/projected/8f8b0d5f-94b6-439a-bbb0-35fb828131dc-kube-api-access-x2vgt\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:01.581727 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.581694 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr"] Nov 03 03:13:01.758024 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:01.757935 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:02.043597 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:02.043521 5463 generic.go:356] "Generic (PLEG): container finished" podID="8f8b0d5f-94b6-439a-bbb0-35fb828131dc" containerID="f03db564fe7b11373e7598a551d7d2fc033384c465a12387536d243ba12b8082" exitCode=0 Nov 03 03:13:02.043597 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:02.043575 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" Nov 03 03:13:02.043786 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:02.043598 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" event={"ID":"8f8b0d5f-94b6-439a-bbb0-35fb828131dc","Type":"ContainerDied","Data":"f03db564fe7b11373e7598a551d7d2fc033384c465a12387536d243ba12b8082"} Nov 03 03:13:02.043786 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:02.043648 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w" event={"ID":"8f8b0d5f-94b6-439a-bbb0-35fb828131dc","Type":"ContainerDied","Data":"3a1c4961c696a147e455fd24ecb613b00c2513583aa973b6313117fa85976d66"} Nov 03 03:13:02.043786 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:02.043669 5463 scope.go:122] "RemoveContainer" containerID="f03db564fe7b11373e7598a551d7d2fc033384c465a12387536d243ba12b8082" Nov 03 03:13:02.056131 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:02.056114 5463 scope.go:122] "RemoveContainer" containerID="fbaeb2432e1d7a5bf0a9433268d8a332624e3c7d5158bfbb054a60d28513baab" Nov 03 03:13:02.067123 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:02.067105 5463 scope.go:122] "RemoveContainer" containerID="f03db564fe7b11373e7598a551d7d2fc033384c465a12387536d243ba12b8082" Nov 03 03:13:02.067411 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:13:02.067393 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f03db564fe7b11373e7598a551d7d2fc033384c465a12387536d243ba12b8082\": container with ID starting with f03db564fe7b11373e7598a551d7d2fc033384c465a12387536d243ba12b8082 not found: ID does not exist" containerID="f03db564fe7b11373e7598a551d7d2fc033384c465a12387536d243ba12b8082" Nov 03 03:13:02.067496 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:02.067423 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f03db564fe7b11373e7598a551d7d2fc033384c465a12387536d243ba12b8082"} err="failed to get container status \"f03db564fe7b11373e7598a551d7d2fc033384c465a12387536d243ba12b8082\": rpc error: code = NotFound desc = could not find container \"f03db564fe7b11373e7598a551d7d2fc033384c465a12387536d243ba12b8082\": container with ID starting with f03db564fe7b11373e7598a551d7d2fc033384c465a12387536d243ba12b8082 not found: ID does not exist" Nov 03 03:13:02.067496 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:02.067448 5463 scope.go:122] "RemoveContainer" containerID="fbaeb2432e1d7a5bf0a9433268d8a332624e3c7d5158bfbb054a60d28513baab" Nov 03 03:13:02.067719 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:13:02.067703 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbaeb2432e1d7a5bf0a9433268d8a332624e3c7d5158bfbb054a60d28513baab\": container with ID starting with fbaeb2432e1d7a5bf0a9433268d8a332624e3c7d5158bfbb054a60d28513baab not found: ID does not exist" containerID="fbaeb2432e1d7a5bf0a9433268d8a332624e3c7d5158bfbb054a60d28513baab" Nov 03 03:13:02.067780 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:02.067729 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbaeb2432e1d7a5bf0a9433268d8a332624e3c7d5158bfbb054a60d28513baab"} err="failed to get container status \"fbaeb2432e1d7a5bf0a9433268d8a332624e3c7d5158bfbb054a60d28513baab\": rpc error: code = NotFound desc = could not find container \"fbaeb2432e1d7a5bf0a9433268d8a332624e3c7d5158bfbb054a60d28513baab\": container with ID starting with fbaeb2432e1d7a5bf0a9433268d8a332624e3c7d5158bfbb054a60d28513baab not found: ID does not exist" Nov 03 03:13:02.209694 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:02.209653 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w"] Nov 03 03:13:02.260322 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:02.260290 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-78b5c5-openshift-default-68ff44956c-plz6w"] Nov 03 03:13:02.442414 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:02.442383 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f8b0d5f-94b6-439a-bbb0-35fb828131dc" path="/var/lib/kubelet/pods/8f8b0d5f-94b6-439a-bbb0-35fb828131dc/volumes" Nov 03 03:13:02.498497 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:02.498463 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:04.536753 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:04.536713 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc" podUID="fcc529d1-9757-467a-bde9-c0afa4822017" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:13:04.537232 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:04.536783 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst" podUID="e01f46d8-b6be-4f51-9291-a67a09771b4e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:13:04.634062 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:04.634030 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:07.346945 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:07.346909 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:07.954023 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:07.953993 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:11.474551 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:11.474467 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:13.006486 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:13:13.006443 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:13:13.242147 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:13.242114 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:13.801775 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:13.801741 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:14.355130 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:14.355097 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl"] Nov 03 03:13:14.460755 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:14.460727 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68"] Nov 03 03:13:14.536036 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:14.536000 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc" podUID="fcc529d1-9757-467a-bde9-c0afa4822017" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:13:14.536216 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:14.536009 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst" podUID="e01f46d8-b6be-4f51-9291-a67a09771b4e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:13:14.746249 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:14.746217 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:14.905686 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:14.905650 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:16.601945 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:16.601906 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" containerID="cri-o://2ffdf6d388c8da0fdeffe1f2ad07410354e18a383ecd90b009bc71408b72d047" gracePeriod=45 Nov 03 03:13:16.860244 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:16.860168 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 03:13:16.931988 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:16.931961 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-model-cache\") pod \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " Nov 03 03:13:16.932132 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:16.932038 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/544033d3-9c0c-4e56-864c-54fcf9d69d7e-kube-api-access-8qw4q\" (UniqueName: \"kubernetes.io/projected/544033d3-9c0c-4e56-864c-54fcf9d69d7e-kube-api-access-8qw4q\") pod \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " Nov 03 03:13:16.932132 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:16.932124 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/544033d3-9c0c-4e56-864c-54fcf9d69d7e-tls-certs\" (UniqueName: \"kubernetes.io/secret/544033d3-9c0c-4e56-864c-54fcf9d69d7e-tls-certs\") pod \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " Nov 03 03:13:16.932294 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:16.932244 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-model-cache" pod "544033d3-9c0c-4e56-864c-54fcf9d69d7e" (UID: "544033d3-9c0c-4e56-864c-54fcf9d69d7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:16.932294 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:16.932245 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-home\" (UniqueName: \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-home\") pod \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " Nov 03 03:13:16.932415 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:16.932297 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-dshm\") pod \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " Nov 03 03:13:16.932415 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:16.932344 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-tmp-dir\") pod \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\" (UID: \"544033d3-9c0c-4e56-864c-54fcf9d69d7e\") " Nov 03 03:13:16.932530 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:16.932459 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-home" pod "544033d3-9c0c-4e56-864c-54fcf9d69d7e" (UID: "544033d3-9c0c-4e56-864c-54fcf9d69d7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:16.932651 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:16.932630 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-tmp-dir" pod "544033d3-9c0c-4e56-864c-54fcf9d69d7e" (UID: "544033d3-9c0c-4e56-864c-54fcf9d69d7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:16.932753 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:16.932735 5463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-tmp-dir\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:16.932798 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:16.932759 5463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-model-cache\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:16.932798 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:16.932774 5463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-home\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:16.934476 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:16.934451 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-dshm" pod "544033d3-9c0c-4e56-864c-54fcf9d69d7e" (UID: "544033d3-9c0c-4e56-864c-54fcf9d69d7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:13:16.934648 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:16.934628 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/544033d3-9c0c-4e56-864c-54fcf9d69d7e-tls-certs" pod "544033d3-9c0c-4e56-864c-54fcf9d69d7e" (UID: "544033d3-9c0c-4e56-864c-54fcf9d69d7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:13:16.934700 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:16.934635 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/544033d3-9c0c-4e56-864c-54fcf9d69d7e-kube-api-access-8qw4q" pod "544033d3-9c0c-4e56-864c-54fcf9d69d7e" (UID: "544033d3-9c0c-4e56-864c-54fcf9d69d7e"). InnerVolumeSpecName "kube-api-access-8qw4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:17.034014 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:17.033967 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qw4q\" (UniqueName: \"kubernetes.io/projected/544033d3-9c0c-4e56-864c-54fcf9d69d7e-kube-api-access-8qw4q\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:17.034014 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:17.034013 5463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/544033d3-9c0c-4e56-864c-54fcf9d69d7e-tls-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:17.034014 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:17.034028 5463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/544033d3-9c0c-4e56-864c-54fcf9d69d7e-dshm\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:17.141323 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:17.141166 5463 generic.go:356] "Generic (PLEG): container finished" podID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerID="2ffdf6d388c8da0fdeffe1f2ad07410354e18a383ecd90b009bc71408b72d047" exitCode=0 Nov 03 03:13:17.141323 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:17.141254 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" event={"ID":"544033d3-9c0c-4e56-864c-54fcf9d69d7e","Type":"ContainerDied","Data":"2ffdf6d388c8da0fdeffe1f2ad07410354e18a383ecd90b009bc71408b72d047"} Nov 03 03:13:17.141323 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:17.141296 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" event={"ID":"544033d3-9c0c-4e56-864c-54fcf9d69d7e","Type":"ContainerDied","Data":"6410b07bb06ed2a3593a520419eafd2c7b407c42872a0b693da9db4d4d850881"} Nov 03 03:13:17.141323 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:17.141313 5463 scope.go:122] "RemoveContainer" containerID="2ffdf6d388c8da0fdeffe1f2ad07410354e18a383ecd90b009bc71408b72d047" Nov 03 03:13:17.141664 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:17.141245 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr" Nov 03 03:13:17.153568 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:17.153547 5463 scope.go:122] "RemoveContainer" containerID="2ffdf6d388c8da0fdeffe1f2ad07410354e18a383ecd90b009bc71408b72d047" Nov 03 03:13:17.153939 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:13:17.153846 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ffdf6d388c8da0fdeffe1f2ad07410354e18a383ecd90b009bc71408b72d047\": container with ID starting with 2ffdf6d388c8da0fdeffe1f2ad07410354e18a383ecd90b009bc71408b72d047 not found: ID does not exist" containerID="2ffdf6d388c8da0fdeffe1f2ad07410354e18a383ecd90b009bc71408b72d047" Nov 03 03:13:17.153939 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:17.153882 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffdf6d388c8da0fdeffe1f2ad07410354e18a383ecd90b009bc71408b72d047"} err="failed to get container status \"2ffdf6d388c8da0fdeffe1f2ad07410354e18a383ecd90b009bc71408b72d047\": rpc error: code = NotFound desc = could not find container \"2ffdf6d388c8da0fdeffe1f2ad07410354e18a383ecd90b009bc71408b72d047\": container with ID starting with 2ffdf6d388c8da0fdeffe1f2ad07410354e18a383ecd90b009bc71408b72d047 not found: ID does not exist" Nov 03 03:13:17.309792 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:17.309758 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr"] Nov 03 03:13:17.311767 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:17.311748 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/premium-w1-1bccbd-kserve-699db8f55b-mkkkr"] Nov 03 03:13:17.677720 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:17.677683 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:17.837463 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:17.837415 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:18.151127 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:18.151101 5463 generic.go:356] "Generic (PLEG): container finished" podID="fcc529d1-9757-467a-bde9-c0afa4822017" containerID="0fefd49e5caa378958ce5bef3a9dc5bdafe9dd5ba2e5dbbc83daa124e0e69fcd" exitCode=0 Nov 03 03:13:18.151283 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:18.151157 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc" event={"ID":"fcc529d1-9757-467a-bde9-c0afa4822017","Type":"ContainerDied","Data":"0fefd49e5caa378958ce5bef3a9dc5bdafe9dd5ba2e5dbbc83daa124e0e69fcd"} Nov 03 03:13:18.279258 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:18.279170 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc" Nov 03 03:13:18.345829 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:18.345802 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcc529d1-9757-467a-bde9-c0afa4822017-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcc529d1-9757-467a-bde9-c0afa4822017-plugins-config-volume\") pod \"fcc529d1-9757-467a-bde9-c0afa4822017\" (UID: \"fcc529d1-9757-467a-bde9-c0afa4822017\") " Nov 03 03:13:18.346000 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:18.345866 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc529d1-9757-467a-bde9-c0afa4822017-kube-api-access-trlct\" (UniqueName: \"kubernetes.io/projected/fcc529d1-9757-467a-bde9-c0afa4822017-kube-api-access-trlct\") pod \"fcc529d1-9757-467a-bde9-c0afa4822017\" (UID: \"fcc529d1-9757-467a-bde9-c0afa4822017\") " Nov 03 03:13:18.346236 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:18.346212 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc529d1-9757-467a-bde9-c0afa4822017-plugins-config-volume" pod "fcc529d1-9757-467a-bde9-c0afa4822017" (UID: "fcc529d1-9757-467a-bde9-c0afa4822017"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:13:18.346348 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:18.346331 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcc529d1-9757-467a-bde9-c0afa4822017-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:18.347915 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:18.347881 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc529d1-9757-467a-bde9-c0afa4822017-kube-api-access-trlct" pod "fcc529d1-9757-467a-bde9-c0afa4822017" (UID: "fcc529d1-9757-467a-bde9-c0afa4822017"). InnerVolumeSpecName "kube-api-access-trlct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:18.364833 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:18.364806 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst" Nov 03 03:13:18.442411 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:18.442372 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" path="/var/lib/kubelet/pods/544033d3-9c0c-4e56-864c-54fcf9d69d7e/volumes" Nov 03 03:13:18.446692 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:18.446662 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e01f46d8-b6be-4f51-9291-a67a09771b4e-kube-api-access-5szv7\" (UniqueName: \"kubernetes.io/projected/e01f46d8-b6be-4f51-9291-a67a09771b4e-kube-api-access-5szv7\") pod \"e01f46d8-b6be-4f51-9291-a67a09771b4e\" (UID: \"e01f46d8-b6be-4f51-9291-a67a09771b4e\") " Nov 03 03:13:18.446865 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:18.446744 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e01f46d8-b6be-4f51-9291-a67a09771b4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e01f46d8-b6be-4f51-9291-a67a09771b4e-plugins-config-volume\") pod \"e01f46d8-b6be-4f51-9291-a67a09771b4e\" (UID: \"e01f46d8-b6be-4f51-9291-a67a09771b4e\") " Nov 03 03:13:18.447279 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:18.447033 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trlct\" (UniqueName: \"kubernetes.io/projected/fcc529d1-9757-467a-bde9-c0afa4822017-kube-api-access-trlct\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:18.447279 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:18.447240 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e01f46d8-b6be-4f51-9291-a67a09771b4e-plugins-config-volume" pod "e01f46d8-b6be-4f51-9291-a67a09771b4e" (UID: "e01f46d8-b6be-4f51-9291-a67a09771b4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:13:18.448898 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:18.448874 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01f46d8-b6be-4f51-9291-a67a09771b4e-kube-api-access-5szv7" pod "e01f46d8-b6be-4f51-9291-a67a09771b4e" (UID: "e01f46d8-b6be-4f51-9291-a67a09771b4e"). InnerVolumeSpecName "kube-api-access-5szv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:18.495038 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:18.495006 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:18.547759 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:18.547673 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5szv7\" (UniqueName: \"kubernetes.io/projected/e01f46d8-b6be-4f51-9291-a67a09771b4e-kube-api-access-5szv7\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:18.547759 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:18.547713 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e01f46d8-b6be-4f51-9291-a67a09771b4e-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:19.158278 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:19.158233 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc" event={"ID":"fcc529d1-9757-467a-bde9-c0afa4822017","Type":"ContainerDied","Data":"7bb7ac40385d916826dd093df6f16155191667976900d80110f679b9ced6c25c"} Nov 03 03:13:19.158278 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:19.158266 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc" Nov 03 03:13:19.158911 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:19.158344 5463 scope.go:122] "RemoveContainer" containerID="0fefd49e5caa378958ce5bef3a9dc5bdafe9dd5ba2e5dbbc83daa124e0e69fcd" Nov 03 03:13:19.159901 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:19.159752 5463 generic.go:356] "Generic (PLEG): container finished" podID="e01f46d8-b6be-4f51-9291-a67a09771b4e" containerID="4f6d5c06c5363b6c00afe8458f67a9b558fe721d19f64abd9563c44bea0c6e5f" exitCode=0 Nov 03 03:13:19.159901 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:19.159826 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst" Nov 03 03:13:19.159901 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:19.159850 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst" event={"ID":"e01f46d8-b6be-4f51-9291-a67a09771b4e","Type":"ContainerDied","Data":"4f6d5c06c5363b6c00afe8458f67a9b558fe721d19f64abd9563c44bea0c6e5f"} Nov 03 03:13:19.160057 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:19.159922 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst" event={"ID":"e01f46d8-b6be-4f51-9291-a67a09771b4e","Type":"ContainerDied","Data":"8b80bdc8d6b88376889b4b8c82b0e3099588ead75dc18ff750003bdbb9a6126b"} Nov 03 03:13:19.169169 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:19.169151 5463 scope.go:122] "RemoveContainer" containerID="4f6d5c06c5363b6c00afe8458f67a9b558fe721d19f64abd9563c44bea0c6e5f" Nov 03 03:13:19.177720 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:19.177704 5463 scope.go:122] "RemoveContainer" containerID="4f6d5c06c5363b6c00afe8458f67a9b558fe721d19f64abd9563c44bea0c6e5f" Nov 03 03:13:19.178005 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:13:19.177984 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f6d5c06c5363b6c00afe8458f67a9b558fe721d19f64abd9563c44bea0c6e5f\": container with ID starting with 4f6d5c06c5363b6c00afe8458f67a9b558fe721d19f64abd9563c44bea0c6e5f not found: ID does not exist" containerID="4f6d5c06c5363b6c00afe8458f67a9b558fe721d19f64abd9563c44bea0c6e5f" Nov 03 03:13:19.178061 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:19.178020 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f6d5c06c5363b6c00afe8458f67a9b558fe721d19f64abd9563c44bea0c6e5f"} err="failed to get container status \"4f6d5c06c5363b6c00afe8458f67a9b558fe721d19f64abd9563c44bea0c6e5f\": rpc error: code = NotFound desc = could not find container \"4f6d5c06c5363b6c00afe8458f67a9b558fe721d19f64abd9563c44bea0c6e5f\": container with ID starting with 4f6d5c06c5363b6c00afe8458f67a9b558fe721d19f64abd9563c44bea0c6e5f not found: ID does not exist" Nov 03 03:13:19.196068 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:19.196050 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:19.327123 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:19.327058 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst"] Nov 03 03:13:19.335733 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:19.335709 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f7d9bb-66d8864fbc-n6pst"] Nov 03 03:13:19.356145 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:19.356117 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl" podUID="160c3088-0ac6-4694-80b9-67537f5be9eb" containerName="payload-processing" containerID="cri-o://6b3e81d8476dbe1fcbead8e795568c71a87f79b7f82ecbe71688f894a306359d" gracePeriod=25 Nov 03 03:13:19.461964 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:19.461930 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68" podUID="fcd015dd-30d1-4602-9c58-e7f12e5193d7" containerName="payload-pre-processing" containerID="cri-o://db6ae0eabc862a6d2b0f2d7daad8ab05880932a1eed1f29eeedfe32ca8f7316f" gracePeriod=25 Nov 03 03:13:19.487641 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:19.487615 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc"] Nov 03 03:13:19.491975 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:19.491955 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f7d9bb-6c76b4cb59-bn2kc"] Nov 03 03:13:20.238858 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:20.238826 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:20.443039 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:20.443004 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e01f46d8-b6be-4f51-9291-a67a09771b4e" path="/var/lib/kubelet/pods/e01f46d8-b6be-4f51-9291-a67a09771b4e/volumes" Nov 03 03:13:20.443624 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:20.443603 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcc529d1-9757-467a-bde9-c0afa4822017" path="/var/lib/kubelet/pods/fcc529d1-9757-467a-bde9-c0afa4822017/volumes" Nov 03 03:13:21.310028 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:21.309987 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl" podUID="160c3088-0ac6-4694-80b9-67537f5be9eb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:13:21.310443 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:21.309992 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68" podUID="fcd015dd-30d1-4602-9c58-e7f12e5193d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:13:21.492031 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:21.492001 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:21.654280 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:21.654248 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:23.017394 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:23.017355 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:23.626247 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:23.626202 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:23.868591 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:23.868560 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:25.854649 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:25.854605 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:26.024772 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:26.024725 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:27.144916 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:27.144879 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb"] Nov 03 03:13:27.160568 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:27.160535 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:27.255065 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:27.255034 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw"] Nov 03 03:13:27.508673 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:27.508575 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:27.664341 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:27.664301 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:29.071068 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:29.071035 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:29.804033 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:29.803993 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:31.310816 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:31.310723 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl" podUID="160c3088-0ac6-4694-80b9-67537f5be9eb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:13:31.311281 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:31.310729 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68" podUID="fcd015dd-30d1-4602-9c58-e7f12e5193d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:13:32.145465 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:32.145425 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb" podUID="699e3934-8850-4132-9f5c-3e446be27484" containerName="payload-processing" containerID="cri-o://f7f40a60734173a7914d3b6f5a264ebf5dd50a16813b847d8f97ab87153f294d" gracePeriod=25 Nov 03 03:13:32.255425 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:32.255387 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw" podUID="edb46670-67e5-4fc2-a94d-87634bc91bc4" containerName="payload-pre-processing" containerID="cri-o://38a35ab810ef53ff3e3baa8b8486b227036a8d8b9f25c7375c0758f00abf57de" gracePeriod=25 Nov 03 03:13:32.780799 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:32.780762 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:33.917801 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:33.917768 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:36.287095 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:36.287056 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:36.448675 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:36.448643 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:37.983243 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:37.983151 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9"] Nov 03 03:13:37.983790 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:37.983409 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" podUID="d6da7ddb-bf04-456f-9821-dfb9d517e0b6" containerName="maas-api" containerID="cri-o://d5b0f3493ee25a38d19d119985fae1178bd1db0c86c0866e427dd5b95176458a" gracePeriod=30 Nov 03 03:13:38.144482 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.144439 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:38.227139 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.226854 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb" podUID="699e3934-8850-4132-9f5c-3e446be27484" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:13:38.227594 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.227523 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw" podUID="edb46670-67e5-4fc2-a94d-87634bc91bc4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:13:38.263057 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.263032 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" Nov 03 03:13:38.291412 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.291375 5463 generic.go:356] "Generic (PLEG): container finished" podID="d6da7ddb-bf04-456f-9821-dfb9d517e0b6" containerID="d5b0f3493ee25a38d19d119985fae1178bd1db0c86c0866e427dd5b95176458a" exitCode=0 Nov 03 03:13:38.291543 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.291415 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" Nov 03 03:13:38.291543 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.291455 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" event={"ID":"d6da7ddb-bf04-456f-9821-dfb9d517e0b6","Type":"ContainerDied","Data":"d5b0f3493ee25a38d19d119985fae1178bd1db0c86c0866e427dd5b95176458a"} Nov 03 03:13:38.291543 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.291517 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9" event={"ID":"d6da7ddb-bf04-456f-9821-dfb9d517e0b6","Type":"ContainerDied","Data":"86f93b06035c56d04e9f030a5c992efdb70fb66a9494210c6d277b9790e74858"} Nov 03 03:13:38.291543 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.291541 5463 scope.go:122] "RemoveContainer" containerID="d5b0f3493ee25a38d19d119985fae1178bd1db0c86c0866e427dd5b95176458a" Nov 03 03:13:38.306063 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.306038 5463 scope.go:122] "RemoveContainer" containerID="d5b0f3493ee25a38d19d119985fae1178bd1db0c86c0866e427dd5b95176458a" Nov 03 03:13:38.306378 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:13:38.306353 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b0f3493ee25a38d19d119985fae1178bd1db0c86c0866e427dd5b95176458a\": container with ID starting with d5b0f3493ee25a38d19d119985fae1178bd1db0c86c0866e427dd5b95176458a not found: ID does not exist" containerID="d5b0f3493ee25a38d19d119985fae1178bd1db0c86c0866e427dd5b95176458a" Nov 03 03:13:38.306460 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.306389 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b0f3493ee25a38d19d119985fae1178bd1db0c86c0866e427dd5b95176458a"} err="failed to get container status \"d5b0f3493ee25a38d19d119985fae1178bd1db0c86c0866e427dd5b95176458a\": rpc error: code = NotFound desc = could not find container \"d5b0f3493ee25a38d19d119985fae1178bd1db0c86c0866e427dd5b95176458a\": container with ID starting with d5b0f3493ee25a38d19d119985fae1178bd1db0c86c0866e427dd5b95176458a not found: ID does not exist" Nov 03 03:13:38.369382 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.369351 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-openshift-service-ca\") pod \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\" (UID: \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\") " Nov 03 03:13:38.369543 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.369459 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-maas-api-tls\") pod \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\" (UID: \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\") " Nov 03 03:13:38.369543 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.369532 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-kube-api-access-2rksj\" (UniqueName: \"kubernetes.io/projected/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-kube-api-access-2rksj\") pod \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\" (UID: \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\") " Nov 03 03:13:38.369989 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.369590 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-metrics-certs\") pod \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\" (UID: \"d6da7ddb-bf04-456f-9821-dfb9d517e0b6\") " Nov 03 03:13:38.369989 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.369730 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-openshift-service-ca" pod "d6da7ddb-bf04-456f-9821-dfb9d517e0b6" (UID: "d6da7ddb-bf04-456f-9821-dfb9d517e0b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:13:38.369989 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.369911 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:38.371680 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.371649 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-maas-api-tls" pod "d6da7ddb-bf04-456f-9821-dfb9d517e0b6" (UID: "d6da7ddb-bf04-456f-9821-dfb9d517e0b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:13:38.371818 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.371795 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-kube-api-access-2rksj" pod "d6da7ddb-bf04-456f-9821-dfb9d517e0b6" (UID: "d6da7ddb-bf04-456f-9821-dfb9d517e0b6"). InnerVolumeSpecName "kube-api-access-2rksj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:38.371879 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.371800 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-metrics-certs" pod "d6da7ddb-bf04-456f-9821-dfb9d517e0b6" (UID: "d6da7ddb-bf04-456f-9821-dfb9d517e0b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:13:38.471294 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.471268 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:38.471294 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.471292 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rksj\" (UniqueName: \"kubernetes.io/projected/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-kube-api-access-2rksj\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:38.471463 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.471302 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6da7ddb-bf04-456f-9821-dfb9d517e0b6-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:38.711225 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.708538 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9"] Nov 03 03:13:38.711225 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.710907 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f735f4-78d8455854-zzkk9"] Nov 03 03:13:38.785153 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.785116 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:38.948040 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:38.947996 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:39.280763 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:39.280728 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:39.645657 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:39.645626 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:39.649333 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:39.649308 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl" Nov 03 03:13:39.722410 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:39.722387 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68" Nov 03 03:13:39.783623 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:39.783590 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160c3088-0ac6-4694-80b9-67537f5be9eb-kube-api-access-fnjtv\" (UniqueName: \"kubernetes.io/projected/160c3088-0ac6-4694-80b9-67537f5be9eb-kube-api-access-fnjtv\") pod \"160c3088-0ac6-4694-80b9-67537f5be9eb\" (UID: \"160c3088-0ac6-4694-80b9-67537f5be9eb\") " Nov 03 03:13:39.783798 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:39.783637 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/160c3088-0ac6-4694-80b9-67537f5be9eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160c3088-0ac6-4694-80b9-67537f5be9eb-plugins-config-volume\") pod \"160c3088-0ac6-4694-80b9-67537f5be9eb\" (UID: \"160c3088-0ac6-4694-80b9-67537f5be9eb\") " Nov 03 03:13:39.784052 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:39.784029 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160c3088-0ac6-4694-80b9-67537f5be9eb-plugins-config-volume" pod "160c3088-0ac6-4694-80b9-67537f5be9eb" (UID: "160c3088-0ac6-4694-80b9-67537f5be9eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:13:39.785731 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:39.785704 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160c3088-0ac6-4694-80b9-67537f5be9eb-kube-api-access-fnjtv" pod "160c3088-0ac6-4694-80b9-67537f5be9eb" (UID: "160c3088-0ac6-4694-80b9-67537f5be9eb"). InnerVolumeSpecName "kube-api-access-fnjtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:39.860739 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:39.860673 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:39.884769 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:39.884745 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcd015dd-30d1-4602-9c58-e7f12e5193d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd015dd-30d1-4602-9c58-e7f12e5193d7-plugins-config-volume\") pod \"fcd015dd-30d1-4602-9c58-e7f12e5193d7\" (UID: \"fcd015dd-30d1-4602-9c58-e7f12e5193d7\") " Nov 03 03:13:39.884915 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:39.884811 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd015dd-30d1-4602-9c58-e7f12e5193d7-kube-api-access-677c6\" (UniqueName: \"kubernetes.io/projected/fcd015dd-30d1-4602-9c58-e7f12e5193d7-kube-api-access-677c6\") pod \"fcd015dd-30d1-4602-9c58-e7f12e5193d7\" (UID: \"fcd015dd-30d1-4602-9c58-e7f12e5193d7\") " Nov 03 03:13:39.885105 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:39.885084 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcd015dd-30d1-4602-9c58-e7f12e5193d7-plugins-config-volume" pod "fcd015dd-30d1-4602-9c58-e7f12e5193d7" (UID: "fcd015dd-30d1-4602-9c58-e7f12e5193d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:13:39.885161 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:39.885105 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnjtv\" (UniqueName: \"kubernetes.io/projected/160c3088-0ac6-4694-80b9-67537f5be9eb-kube-api-access-fnjtv\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:39.885161 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:39.885122 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160c3088-0ac6-4694-80b9-67537f5be9eb-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:39.886999 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:39.886972 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd015dd-30d1-4602-9c58-e7f12e5193d7-kube-api-access-677c6" pod "fcd015dd-30d1-4602-9c58-e7f12e5193d7" (UID: "fcd015dd-30d1-4602-9c58-e7f12e5193d7"). InnerVolumeSpecName "kube-api-access-677c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:39.985907 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:39.985884 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd015dd-30d1-4602-9c58-e7f12e5193d7-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:39.985907 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:39.985907 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-677c6\" (UniqueName: \"kubernetes.io/projected/fcd015dd-30d1-4602-9c58-e7f12e5193d7-kube-api-access-677c6\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:40.305727 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:40.305690 5463 generic.go:356] "Generic (PLEG): container finished" podID="160c3088-0ac6-4694-80b9-67537f5be9eb" containerID="6b3e81d8476dbe1fcbead8e795568c71a87f79b7f82ecbe71688f894a306359d" exitCode=0 Nov 03 03:13:40.306163 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:40.305755 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl" Nov 03 03:13:40.306163 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:40.305774 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl" event={"ID":"160c3088-0ac6-4694-80b9-67537f5be9eb","Type":"ContainerDied","Data":"6b3e81d8476dbe1fcbead8e795568c71a87f79b7f82ecbe71688f894a306359d"} Nov 03 03:13:40.306163 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:40.305855 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl" event={"ID":"160c3088-0ac6-4694-80b9-67537f5be9eb","Type":"ContainerDied","Data":"ebe58db55413513ce7b2c90e2e8c8471bc373111909df54d8dd28b88336c821b"} Nov 03 03:13:40.306163 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:40.305877 5463 scope.go:122] "RemoveContainer" containerID="6b3e81d8476dbe1fcbead8e795568c71a87f79b7f82ecbe71688f894a306359d" Nov 03 03:13:40.307420 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:40.307397 5463 generic.go:356] "Generic (PLEG): container finished" podID="fcd015dd-30d1-4602-9c58-e7f12e5193d7" containerID="db6ae0eabc862a6d2b0f2d7daad8ab05880932a1eed1f29eeedfe32ca8f7316f" exitCode=0 Nov 03 03:13:40.307540 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:40.307449 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68" event={"ID":"fcd015dd-30d1-4602-9c58-e7f12e5193d7","Type":"ContainerDied","Data":"db6ae0eabc862a6d2b0f2d7daad8ab05880932a1eed1f29eeedfe32ca8f7316f"} Nov 03 03:13:40.307540 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:40.307455 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68" Nov 03 03:13:40.307540 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:40.307488 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68" event={"ID":"fcd015dd-30d1-4602-9c58-e7f12e5193d7","Type":"ContainerDied","Data":"8d8e3927e50514cb999eb643958b0b5ec2a95ff1a45a07ac4b937818138e7de5"} Nov 03 03:13:40.316312 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:40.316293 5463 scope.go:122] "RemoveContainer" containerID="6b3e81d8476dbe1fcbead8e795568c71a87f79b7f82ecbe71688f894a306359d" Nov 03 03:13:40.316646 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:13:40.316623 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b3e81d8476dbe1fcbead8e795568c71a87f79b7f82ecbe71688f894a306359d\": container with ID starting with 6b3e81d8476dbe1fcbead8e795568c71a87f79b7f82ecbe71688f894a306359d not found: ID does not exist" containerID="6b3e81d8476dbe1fcbead8e795568c71a87f79b7f82ecbe71688f894a306359d" Nov 03 03:13:40.316752 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:40.316651 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3e81d8476dbe1fcbead8e795568c71a87f79b7f82ecbe71688f894a306359d"} err="failed to get container status \"6b3e81d8476dbe1fcbead8e795568c71a87f79b7f82ecbe71688f894a306359d\": rpc error: code = NotFound desc = could not find container \"6b3e81d8476dbe1fcbead8e795568c71a87f79b7f82ecbe71688f894a306359d\": container with ID starting with 6b3e81d8476dbe1fcbead8e795568c71a87f79b7f82ecbe71688f894a306359d not found: ID does not exist" Nov 03 03:13:40.316752 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:40.316674 5463 scope.go:122] "RemoveContainer" containerID="db6ae0eabc862a6d2b0f2d7daad8ab05880932a1eed1f29eeedfe32ca8f7316f" Nov 03 03:13:40.325012 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:40.324991 5463 scope.go:122] "RemoveContainer" containerID="db6ae0eabc862a6d2b0f2d7daad8ab05880932a1eed1f29eeedfe32ca8f7316f" Nov 03 03:13:40.325285 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:13:40.325265 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db6ae0eabc862a6d2b0f2d7daad8ab05880932a1eed1f29eeedfe32ca8f7316f\": container with ID starting with db6ae0eabc862a6d2b0f2d7daad8ab05880932a1eed1f29eeedfe32ca8f7316f not found: ID does not exist" containerID="db6ae0eabc862a6d2b0f2d7daad8ab05880932a1eed1f29eeedfe32ca8f7316f" Nov 03 03:13:40.325342 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:40.325293 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db6ae0eabc862a6d2b0f2d7daad8ab05880932a1eed1f29eeedfe32ca8f7316f"} err="failed to get container status \"db6ae0eabc862a6d2b0f2d7daad8ab05880932a1eed1f29eeedfe32ca8f7316f\": rpc error: code = NotFound desc = could not find container \"db6ae0eabc862a6d2b0f2d7daad8ab05880932a1eed1f29eeedfe32ca8f7316f\": container with ID starting with db6ae0eabc862a6d2b0f2d7daad8ab05880932a1eed1f29eeedfe32ca8f7316f not found: ID does not exist" Nov 03 03:13:40.442567 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:40.442537 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6da7ddb-bf04-456f-9821-dfb9d517e0b6" path="/var/lib/kubelet/pods/d6da7ddb-bf04-456f-9821-dfb9d517e0b6/volumes" Nov 03 03:13:40.472880 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:40.472847 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68"] Nov 03 03:13:40.522930 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:40.522898 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1bccbd-b545ddfc5-5vg68"] Nov 03 03:13:40.631666 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:40.631583 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl"] Nov 03 03:13:40.634312 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:40.634282 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1bccbd-58c7585cf5-n8sdl"] Nov 03 03:13:41.039881 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:41.039844 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:41.627733 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:41.627698 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:41.791554 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:41.791509 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:42.441649 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:42.441618 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="160c3088-0ac6-4694-80b9-67537f5be9eb" path="/var/lib/kubelet/pods/160c3088-0ac6-4694-80b9-67537f5be9eb/volumes" Nov 03 03:13:42.442044 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:42.442029 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcd015dd-30d1-4602-9c58-e7f12e5193d7" path="/var/lib/kubelet/pods/fcd015dd-30d1-4602-9c58-e7f12e5193d7/volumes" Nov 03 03:13:43.009007 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:13:43.008973 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:13:43.138988 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:43.138952 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:43.653223 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:43.653173 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:45.439668 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:45.439637 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:46.122477 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:46.122442 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:48.212657 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:48.212615 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw" podUID="edb46670-67e5-4fc2-a94d-87634bc91bc4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:13:48.213057 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:48.212850 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb" podUID="699e3934-8850-4132-9f5c-3e446be27484" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:13:50.460976 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:50.460942 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:50.611748 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:50.611715 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:50.764463 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:50.764389 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:51.243578 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:51.243544 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:51.426270 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:51.426239 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:51.587135 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:51.587003 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:52.382225 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:52.382168 5463 generic.go:356] "Generic (PLEG): container finished" podID="edb46670-67e5-4fc2-a94d-87634bc91bc4" containerID="38a35ab810ef53ff3e3baa8b8486b227036a8d8b9f25c7375c0758f00abf57de" exitCode=0 Nov 03 03:13:52.382390 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:52.382238 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw" event={"ID":"edb46670-67e5-4fc2-a94d-87634bc91bc4","Type":"ContainerDied","Data":"38a35ab810ef53ff3e3baa8b8486b227036a8d8b9f25c7375c0758f00abf57de"} Nov 03 03:13:52.384611 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:52.384584 5463 generic.go:356] "Generic (PLEG): container finished" podID="699e3934-8850-4132-9f5c-3e446be27484" containerID="f7f40a60734173a7914d3b6f5a264ebf5dd50a16813b847d8f97ab87153f294d" exitCode=0 Nov 03 03:13:52.384725 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:52.384661 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb" event={"ID":"699e3934-8850-4132-9f5c-3e446be27484","Type":"ContainerDied","Data":"f7f40a60734173a7914d3b6f5a264ebf5dd50a16813b847d8f97ab87153f294d"} Nov 03 03:13:52.408540 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:52.408489 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb" Nov 03 03:13:52.500166 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:52.500140 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/699e3934-8850-4132-9f5c-3e446be27484-kube-api-access-lb4cf\" (UniqueName: \"kubernetes.io/projected/699e3934-8850-4132-9f5c-3e446be27484-kube-api-access-lb4cf\") pod \"699e3934-8850-4132-9f5c-3e446be27484\" (UID: \"699e3934-8850-4132-9f5c-3e446be27484\") " Nov 03 03:13:52.500300 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:52.500219 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/699e3934-8850-4132-9f5c-3e446be27484-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/699e3934-8850-4132-9f5c-3e446be27484-plugins-config-volume\") pod \"699e3934-8850-4132-9f5c-3e446be27484\" (UID: \"699e3934-8850-4132-9f5c-3e446be27484\") " Nov 03 03:13:52.500871 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:52.500842 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/699e3934-8850-4132-9f5c-3e446be27484-plugins-config-volume" pod "699e3934-8850-4132-9f5c-3e446be27484" (UID: "699e3934-8850-4132-9f5c-3e446be27484"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:13:52.510220 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:52.510169 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699e3934-8850-4132-9f5c-3e446be27484-kube-api-access-lb4cf" pod "699e3934-8850-4132-9f5c-3e446be27484" (UID: "699e3934-8850-4132-9f5c-3e446be27484"). InnerVolumeSpecName "kube-api-access-lb4cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:52.517607 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:52.517587 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw" Nov 03 03:13:52.602019 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:52.601982 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edb46670-67e5-4fc2-a94d-87634bc91bc4-kube-api-access-p2qnj\" (UniqueName: \"kubernetes.io/projected/edb46670-67e5-4fc2-a94d-87634bc91bc4-kube-api-access-p2qnj\") pod \"edb46670-67e5-4fc2-a94d-87634bc91bc4\" (UID: \"edb46670-67e5-4fc2-a94d-87634bc91bc4\") " Nov 03 03:13:52.602522 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:52.602073 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edb46670-67e5-4fc2-a94d-87634bc91bc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edb46670-67e5-4fc2-a94d-87634bc91bc4-plugins-config-volume\") pod \"edb46670-67e5-4fc2-a94d-87634bc91bc4\" (UID: \"edb46670-67e5-4fc2-a94d-87634bc91bc4\") " Nov 03 03:13:52.602522 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:52.602413 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb4cf\" (UniqueName: \"kubernetes.io/projected/699e3934-8850-4132-9f5c-3e446be27484-kube-api-access-lb4cf\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:52.602522 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:52.602434 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/699e3934-8850-4132-9f5c-3e446be27484-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:52.602522 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:52.602473 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb46670-67e5-4fc2-a94d-87634bc91bc4-plugins-config-volume" pod "edb46670-67e5-4fc2-a94d-87634bc91bc4" (UID: "edb46670-67e5-4fc2-a94d-87634bc91bc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:13:52.604248 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:52.604226 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb46670-67e5-4fc2-a94d-87634bc91bc4-kube-api-access-p2qnj" pod "edb46670-67e5-4fc2-a94d-87634bc91bc4" (UID: "edb46670-67e5-4fc2-a94d-87634bc91bc4"). InnerVolumeSpecName "kube-api-access-p2qnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:52.703418 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:52.703338 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2qnj\" (UniqueName: \"kubernetes.io/projected/edb46670-67e5-4fc2-a94d-87634bc91bc4-kube-api-access-p2qnj\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:52.703418 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:52.703367 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edb46670-67e5-4fc2-a94d-87634bc91bc4-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:53.392101 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:53.392068 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw" Nov 03 03:13:53.392322 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:53.392095 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw" event={"ID":"edb46670-67e5-4fc2-a94d-87634bc91bc4","Type":"ContainerDied","Data":"153cbd9a5474d26b6ba7badecec6155caea6bbf4467dc5895a28f5db7177c0ed"} Nov 03 03:13:53.392322 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:53.392183 5463 scope.go:122] "RemoveContainer" containerID="38a35ab810ef53ff3e3baa8b8486b227036a8d8b9f25c7375c0758f00abf57de" Nov 03 03:13:53.393571 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:53.393546 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb" event={"ID":"699e3934-8850-4132-9f5c-3e446be27484","Type":"ContainerDied","Data":"0e4eac82bc2710c7f14c3b466e0addcc9f53858070da24b7a3563866d95441af"} Nov 03 03:13:53.393672 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:53.393598 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb" Nov 03 03:13:53.402898 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:53.402883 5463 scope.go:122] "RemoveContainer" containerID="f7f40a60734173a7914d3b6f5a264ebf5dd50a16813b847d8f97ab87153f294d" Nov 03 03:13:53.556874 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:53.556845 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb"] Nov 03 03:13:53.559485 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:53.559464 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f735f4-7f788b5b55-ch4tb"] Nov 03 03:13:53.669164 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:53.669100 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw"] Nov 03 03:13:53.670762 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:53.670743 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f735f4-78f58bc8bc-dcktw"] Nov 03 03:13:54.441747 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:54.441715 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="699e3934-8850-4132-9f5c-3e446be27484" path="/var/lib/kubelet/pods/699e3934-8850-4132-9f5c-3e446be27484/volumes" Nov 03 03:13:54.442110 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:54.442097 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edb46670-67e5-4fc2-a94d-87634bc91bc4" path="/var/lib/kubelet/pods/edb46670-67e5-4fc2-a94d-87634bc91bc4/volumes" Nov 03 03:13:56.341109 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:56.341079 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:56.846086 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:56.846054 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh"] Nov 03 03:13:57.269790 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:57.269758 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:57.421351 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:57.421302 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:57.676723 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:13:57.676691 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:01.327070 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:01.327035 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:01.847247 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:01.847180 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh" podUID="e48e80cd-9263-4d82-8028-2680b548201b" containerName="payload-pre-processing" containerID="cri-o://e54f8e9f650415c3130ebdd2f559c15c276282461ce7e5d774168bb9f1dc7bd2" gracePeriod=25 Nov 03 03:14:03.119789 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:03.119758 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:05.344393 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:05.344351 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:06.043555 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:06.043518 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:07.838819 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:07.838782 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh" podUID="e48e80cd-9263-4d82-8028-2680b548201b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:14:10.255513 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:10.255477 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:12.026205 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:12.026164 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:13.006587 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:14:13.006550 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:14:16.735016 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:16.734967 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:17.236008 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:17.235967 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:17.429389 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:17.429351 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:17.838277 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:17.838242 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh" podUID="e48e80cd-9263-4d82-8028-2680b548201b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:14:18.808540 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:18.808505 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:19.297568 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:19.297530 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78"] Nov 03 03:14:19.298221 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:19.297810 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" podUID="96657503-568c-4ac5-a39c-8abd8eadd843" containerName="istio-proxy" containerID="cri-o://831eb67657d07363ad876d9bcccb3dfb16fcc0e8a7e15edc080fccb64f59212c" gracePeriod=30 Nov 03 03:14:19.312568 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:19.312538 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:19.807907 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:19.807866 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:21.515622 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:21.515588 5463 patch_prober.go:29] interesting pod/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:14:21.516082 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:21.515653 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" podUID="96657503-568c-4ac5-a39c-8abd8eadd843" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:14:21.867479 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:21.867450 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:22.098920 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:22.098897 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh" Nov 03 03:14:22.286786 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:22.286760 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e48e80cd-9263-4d82-8028-2680b548201b-kube-api-access-trkck\" (UniqueName: \"kubernetes.io/projected/e48e80cd-9263-4d82-8028-2680b548201b-kube-api-access-trkck\") pod \"e48e80cd-9263-4d82-8028-2680b548201b\" (UID: \"e48e80cd-9263-4d82-8028-2680b548201b\") " Nov 03 03:14:22.286995 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:22.286839 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e48e80cd-9263-4d82-8028-2680b548201b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48e80cd-9263-4d82-8028-2680b548201b-plugins-config-volume\") pod \"e48e80cd-9263-4d82-8028-2680b548201b\" (UID: \"e48e80cd-9263-4d82-8028-2680b548201b\") " Nov 03 03:14:22.287330 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:22.287298 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e48e80cd-9263-4d82-8028-2680b548201b-plugins-config-volume" pod "e48e80cd-9263-4d82-8028-2680b548201b" (UID: "e48e80cd-9263-4d82-8028-2680b548201b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:14:22.288750 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:22.288718 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48e80cd-9263-4d82-8028-2680b548201b-kube-api-access-trkck" pod "e48e80cd-9263-4d82-8028-2680b548201b" (UID: "e48e80cd-9263-4d82-8028-2680b548201b"). InnerVolumeSpecName "kube-api-access-trkck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:14:22.387863 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:22.387830 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trkck\" (UniqueName: \"kubernetes.io/projected/e48e80cd-9263-4d82-8028-2680b548201b-kube-api-access-trkck\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:22.387863 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:22.387862 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48e80cd-9263-4d82-8028-2680b548201b-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:22.578958 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:22.578798 5463 generic.go:356] "Generic (PLEG): container finished" podID="e48e80cd-9263-4d82-8028-2680b548201b" containerID="e54f8e9f650415c3130ebdd2f559c15c276282461ce7e5d774168bb9f1dc7bd2" exitCode=0 Nov 03 03:14:22.578958 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:22.578861 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh" Nov 03 03:14:22.578958 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:22.578887 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh" event={"ID":"e48e80cd-9263-4d82-8028-2680b548201b","Type":"ContainerDied","Data":"e54f8e9f650415c3130ebdd2f559c15c276282461ce7e5d774168bb9f1dc7bd2"} Nov 03 03:14:22.578958 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:22.578933 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh" event={"ID":"e48e80cd-9263-4d82-8028-2680b548201b","Type":"ContainerDied","Data":"8fe1f2991a0aaed222b7abed283aa162626989165c1ea0f9e63bf0f6f3074f67"} Nov 03 03:14:22.578958 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:22.578954 5463 scope.go:122] "RemoveContainer" containerID="e54f8e9f650415c3130ebdd2f559c15c276282461ce7e5d774168bb9f1dc7bd2" Nov 03 03:14:22.589372 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:22.589349 5463 scope.go:122] "RemoveContainer" containerID="e54f8e9f650415c3130ebdd2f559c15c276282461ce7e5d774168bb9f1dc7bd2" Nov 03 03:14:22.589691 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:14:22.589671 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e54f8e9f650415c3130ebdd2f559c15c276282461ce7e5d774168bb9f1dc7bd2\": container with ID starting with e54f8e9f650415c3130ebdd2f559c15c276282461ce7e5d774168bb9f1dc7bd2 not found: ID does not exist" containerID="e54f8e9f650415c3130ebdd2f559c15c276282461ce7e5d774168bb9f1dc7bd2" Nov 03 03:14:22.589785 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:22.589698 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e54f8e9f650415c3130ebdd2f559c15c276282461ce7e5d774168bb9f1dc7bd2"} err="failed to get container status \"e54f8e9f650415c3130ebdd2f559c15c276282461ce7e5d774168bb9f1dc7bd2\": rpc error: code = NotFound desc = could not find container \"e54f8e9f650415c3130ebdd2f559c15c276282461ce7e5d774168bb9f1dc7bd2\": container with ID starting with e54f8e9f650415c3130ebdd2f559c15c276282461ce7e5d774168bb9f1dc7bd2 not found: ID does not exist" Nov 03 03:14:22.777527 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:22.777487 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh"] Nov 03 03:14:22.782613 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:22.782587 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-957b33-84f45f699-bkwxh"] Nov 03 03:14:24.441296 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.441266 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e48e80cd-9263-4d82-8028-2680b548201b" path="/var/lib/kubelet/pods/e48e80cd-9263-4d82-8028-2680b548201b/volumes" Nov 03 03:14:24.560216 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.560176 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 03:14:24.592567 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.592502 5463 generic.go:356] "Generic (PLEG): container finished" podID="96657503-568c-4ac5-a39c-8abd8eadd843" containerID="831eb67657d07363ad876d9bcccb3dfb16fcc0e8a7e15edc080fccb64f59212c" exitCode=0 Nov 03 03:14:24.592567 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.592554 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" Nov 03 03:14:24.592721 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.592591 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" event={"ID":"96657503-568c-4ac5-a39c-8abd8eadd843","Type":"ContainerDied","Data":"831eb67657d07363ad876d9bcccb3dfb16fcc0e8a7e15edc080fccb64f59212c"} Nov 03 03:14:24.592721 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.592647 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78" event={"ID":"96657503-568c-4ac5-a39c-8abd8eadd843","Type":"ContainerDied","Data":"aed9b2baa11219d85aac6c78fe54a68859177f1340219547895832e6ed9ae347"} Nov 03 03:14:24.592721 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.592678 5463 scope.go:122] "RemoveContainer" containerID="831eb67657d07363ad876d9bcccb3dfb16fcc0e8a7e15edc080fccb64f59212c" Nov 03 03:14:24.601916 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.601894 5463 scope.go:122] "RemoveContainer" containerID="6241213366f593fb3ddb30fed57fd1ca1218fe0cfe79ad2af59c95ecd85f12fc" Nov 03 03:14:24.610296 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.610277 5463 scope.go:122] "RemoveContainer" containerID="831eb67657d07363ad876d9bcccb3dfb16fcc0e8a7e15edc080fccb64f59212c" Nov 03 03:14:24.610529 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:14:24.610513 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"831eb67657d07363ad876d9bcccb3dfb16fcc0e8a7e15edc080fccb64f59212c\": container with ID starting with 831eb67657d07363ad876d9bcccb3dfb16fcc0e8a7e15edc080fccb64f59212c not found: ID does not exist" containerID="831eb67657d07363ad876d9bcccb3dfb16fcc0e8a7e15edc080fccb64f59212c" Nov 03 03:14:24.610583 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.610538 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"831eb67657d07363ad876d9bcccb3dfb16fcc0e8a7e15edc080fccb64f59212c"} err="failed to get container status \"831eb67657d07363ad876d9bcccb3dfb16fcc0e8a7e15edc080fccb64f59212c\": rpc error: code = NotFound desc = could not find container \"831eb67657d07363ad876d9bcccb3dfb16fcc0e8a7e15edc080fccb64f59212c\": container with ID starting with 831eb67657d07363ad876d9bcccb3dfb16fcc0e8a7e15edc080fccb64f59212c not found: ID does not exist" Nov 03 03:14:24.610583 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.610555 5463 scope.go:122] "RemoveContainer" containerID="6241213366f593fb3ddb30fed57fd1ca1218fe0cfe79ad2af59c95ecd85f12fc" Nov 03 03:14:24.610774 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:14:24.610755 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6241213366f593fb3ddb30fed57fd1ca1218fe0cfe79ad2af59c95ecd85f12fc\": container with ID starting with 6241213366f593fb3ddb30fed57fd1ca1218fe0cfe79ad2af59c95ecd85f12fc not found: ID does not exist" containerID="6241213366f593fb3ddb30fed57fd1ca1218fe0cfe79ad2af59c95ecd85f12fc" Nov 03 03:14:24.610851 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.610787 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6241213366f593fb3ddb30fed57fd1ca1218fe0cfe79ad2af59c95ecd85f12fc"} err="failed to get container status \"6241213366f593fb3ddb30fed57fd1ca1218fe0cfe79ad2af59c95ecd85f12fc\": rpc error: code = NotFound desc = could not find container \"6241213366f593fb3ddb30fed57fd1ca1218fe0cfe79ad2af59c95ecd85f12fc\": container with ID starting with 6241213366f593fb3ddb30fed57fd1ca1218fe0cfe79ad2af59c95ecd85f12fc not found: ID does not exist" Nov 03 03:14:24.612034 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.612014 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96657503-568c-4ac5-a39c-8abd8eadd843-kube-api-access-mx7fc\" (UniqueName: \"kubernetes.io/projected/96657503-568c-4ac5-a39c-8abd8eadd843-kube-api-access-mx7fc\") pod \"96657503-568c-4ac5-a39c-8abd8eadd843\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " Nov 03 03:14:24.612108 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.612066 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-istio-data\") pod \"96657503-568c-4ac5-a39c-8abd8eadd843\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " Nov 03 03:14:24.612108 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.612100 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96657503-568c-4ac5-a39c-8abd8eadd843-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96657503-568c-4ac5-a39c-8abd8eadd843-istiod-ca-cert\") pod \"96657503-568c-4ac5-a39c-8abd8eadd843\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " Nov 03 03:14:24.612218 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.612177 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-credential-socket\") pod \"96657503-568c-4ac5-a39c-8abd8eadd843\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " Nov 03 03:14:24.612285 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.612226 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96657503-568c-4ac5-a39c-8abd8eadd843-istio-token\" (UniqueName: \"kubernetes.io/projected/96657503-568c-4ac5-a39c-8abd8eadd843-istio-token\") pod \"96657503-568c-4ac5-a39c-8abd8eadd843\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " Nov 03 03:14:24.612285 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.612257 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-workload-certs\") pod \"96657503-568c-4ac5-a39c-8abd8eadd843\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " Nov 03 03:14:24.612386 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.612294 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-istio-envoy\") pod \"96657503-568c-4ac5-a39c-8abd8eadd843\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " Nov 03 03:14:24.612435 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.612398 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-workload-socket\") pod \"96657503-568c-4ac5-a39c-8abd8eadd843\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " Nov 03 03:14:24.612479 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.612457 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96657503-568c-4ac5-a39c-8abd8eadd843-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96657503-568c-4ac5-a39c-8abd8eadd843-istio-podinfo\") pod \"96657503-568c-4ac5-a39c-8abd8eadd843\" (UID: \"96657503-568c-4ac5-a39c-8abd8eadd843\") " Nov 03 03:14:24.612479 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.612458 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-istio-data" pod "96657503-568c-4ac5-a39c-8abd8eadd843" (UID: "96657503-568c-4ac5-a39c-8abd8eadd843"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:14:24.612560 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.612541 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96657503-568c-4ac5-a39c-8abd8eadd843-istiod-ca-cert" pod "96657503-568c-4ac5-a39c-8abd8eadd843" (UID: "96657503-568c-4ac5-a39c-8abd8eadd843"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:14:24.613003 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.612678 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-workload-certs" pod "96657503-568c-4ac5-a39c-8abd8eadd843" (UID: "96657503-568c-4ac5-a39c-8abd8eadd843"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:14:24.613003 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.612782 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-credential-socket" pod "96657503-568c-4ac5-a39c-8abd8eadd843" (UID: "96657503-568c-4ac5-a39c-8abd8eadd843"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:14:24.613003 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.612924 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:24.613003 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.612942 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96657503-568c-4ac5-a39c-8abd8eadd843-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:24.613003 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.612958 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:24.613003 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.612974 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:24.613003 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.612972 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-workload-socket" pod "96657503-568c-4ac5-a39c-8abd8eadd843" (UID: "96657503-568c-4ac5-a39c-8abd8eadd843"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:14:24.614607 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.614583 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96657503-568c-4ac5-a39c-8abd8eadd843-istio-podinfo" pod "96657503-568c-4ac5-a39c-8abd8eadd843" (UID: "96657503-568c-4ac5-a39c-8abd8eadd843"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:14:24.614905 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.614884 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-istio-envoy" pod "96657503-568c-4ac5-a39c-8abd8eadd843" (UID: "96657503-568c-4ac5-a39c-8abd8eadd843"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:14:24.615084 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.615066 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96657503-568c-4ac5-a39c-8abd8eadd843-kube-api-access-mx7fc" pod "96657503-568c-4ac5-a39c-8abd8eadd843" (UID: "96657503-568c-4ac5-a39c-8abd8eadd843"). InnerVolumeSpecName "kube-api-access-mx7fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:14:24.615134 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.615093 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96657503-568c-4ac5-a39c-8abd8eadd843-istio-token" pod "96657503-568c-4ac5-a39c-8abd8eadd843" (UID: "96657503-568c-4ac5-a39c-8abd8eadd843"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:14:24.713556 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.713530 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96657503-568c-4ac5-a39c-8abd8eadd843-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:24.713556 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.713553 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx7fc\" (UniqueName: \"kubernetes.io/projected/96657503-568c-4ac5-a39c-8abd8eadd843-kube-api-access-mx7fc\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:24.713729 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.713564 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96657503-568c-4ac5-a39c-8abd8eadd843-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:24.713729 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.713573 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:24.713729 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.713582 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96657503-568c-4ac5-a39c-8abd8eadd843-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:24.780613 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.780566 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k"] Nov 03 03:14:24.863108 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:24.863022 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:25.014893 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:25.014859 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78"] Nov 03 03:14:25.018660 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:25.018625 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-957b33-openshift-default-64f9978df9-p7v78"] Nov 03 03:14:26.442809 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:26.442767 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96657503-568c-4ac5-a39c-8abd8eadd843" path="/var/lib/kubelet/pods/96657503-568c-4ac5-a39c-8abd8eadd843/volumes" Nov 03 03:14:26.732117 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:26.732025 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:26.885350 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:26.885321 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:28.443540 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:28.443509 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:28.608419 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:28.608387 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:30.125672 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:30.125639 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:30.281986 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:30.281946 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:31.531319 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:31.531285 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:32.797425 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:32.797388 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:33.274543 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:33.274507 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c"] Nov 03 03:14:33.383086 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:33.383055 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn"] Nov 03 03:14:33.651533 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:33.651498 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:33.810329 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:33.810295 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:36.439853 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:36.439816 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:38.275429 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.275375 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c" podUID="6c5f16c6-739d-4bc0-af05-4587087cc0be" containerName="payload-processing" containerID="cri-o://6996ee59b386b56462e63ecb8f0545da95506e59f9b4cf95565eba49e7cb8190" gracePeriod=25 Nov 03 03:14:38.383662 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.383615 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn" podUID="462caa1d-f10d-4355-ac46-cf9ee0153d23" containerName="payload-pre-processing" containerID="cri-o://8458c21e0225ece4dfd00c018dbe6aeac775d6d6970438db5b4c3de8c05df006" gracePeriod=25 Nov 03 03:14:38.548489 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.548455 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c" Nov 03 03:14:38.572476 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.572436 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:38.633182 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.633160 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn" Nov 03 03:14:38.662556 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.662525 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c5f16c6-739d-4bc0-af05-4587087cc0be-kube-api-access-lhsfw\" (UniqueName: \"kubernetes.io/projected/6c5f16c6-739d-4bc0-af05-4587087cc0be-kube-api-access-lhsfw\") pod \"6c5f16c6-739d-4bc0-af05-4587087cc0be\" (UID: \"6c5f16c6-739d-4bc0-af05-4587087cc0be\") " Nov 03 03:14:38.663401 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.662625 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c5f16c6-739d-4bc0-af05-4587087cc0be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c5f16c6-739d-4bc0-af05-4587087cc0be-plugins-config-volume\") pod \"6c5f16c6-739d-4bc0-af05-4587087cc0be\" (UID: \"6c5f16c6-739d-4bc0-af05-4587087cc0be\") " Nov 03 03:14:38.663401 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.663309 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c5f16c6-739d-4bc0-af05-4587087cc0be-plugins-config-volume" pod "6c5f16c6-739d-4bc0-af05-4587087cc0be" (UID: "6c5f16c6-739d-4bc0-af05-4587087cc0be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:14:38.666317 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.666295 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c5f16c6-739d-4bc0-af05-4587087cc0be-kube-api-access-lhsfw" pod "6c5f16c6-739d-4bc0-af05-4587087cc0be" (UID: "6c5f16c6-739d-4bc0-af05-4587087cc0be"). InnerVolumeSpecName "kube-api-access-lhsfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:14:38.673890 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.673861 5463 generic.go:356] "Generic (PLEG): container finished" podID="6c5f16c6-739d-4bc0-af05-4587087cc0be" containerID="6996ee59b386b56462e63ecb8f0545da95506e59f9b4cf95565eba49e7cb8190" exitCode=0 Nov 03 03:14:38.674027 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.673910 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c" Nov 03 03:14:38.674027 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.673949 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c" event={"ID":"6c5f16c6-739d-4bc0-af05-4587087cc0be","Type":"ContainerDied","Data":"6996ee59b386b56462e63ecb8f0545da95506e59f9b4cf95565eba49e7cb8190"} Nov 03 03:14:38.674027 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.674023 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c" event={"ID":"6c5f16c6-739d-4bc0-af05-4587087cc0be","Type":"ContainerDied","Data":"40001a729609402088f8af92019fc94b13f96831f5ea7452a3037e750762dd03"} Nov 03 03:14:38.674212 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.674044 5463 scope.go:122] "RemoveContainer" containerID="6996ee59b386b56462e63ecb8f0545da95506e59f9b4cf95565eba49e7cb8190" Nov 03 03:14:38.676081 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.676057 5463 generic.go:356] "Generic (PLEG): container finished" podID="462caa1d-f10d-4355-ac46-cf9ee0153d23" containerID="8458c21e0225ece4dfd00c018dbe6aeac775d6d6970438db5b4c3de8c05df006" exitCode=0 Nov 03 03:14:38.676218 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.676111 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn" Nov 03 03:14:38.676218 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.676159 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn" event={"ID":"462caa1d-f10d-4355-ac46-cf9ee0153d23","Type":"ContainerDied","Data":"8458c21e0225ece4dfd00c018dbe6aeac775d6d6970438db5b4c3de8c05df006"} Nov 03 03:14:38.676362 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.676226 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn" event={"ID":"462caa1d-f10d-4355-ac46-cf9ee0153d23","Type":"ContainerDied","Data":"822c8ccac0f14308c7d575f988788b1b9ce96be99a71c4be33ade849b7980f86"} Nov 03 03:14:38.683944 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.683925 5463 scope.go:122] "RemoveContainer" containerID="6996ee59b386b56462e63ecb8f0545da95506e59f9b4cf95565eba49e7cb8190" Nov 03 03:14:38.684600 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:14:38.684277 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6996ee59b386b56462e63ecb8f0545da95506e59f9b4cf95565eba49e7cb8190\": container with ID starting with 6996ee59b386b56462e63ecb8f0545da95506e59f9b4cf95565eba49e7cb8190 not found: ID does not exist" containerID="6996ee59b386b56462e63ecb8f0545da95506e59f9b4cf95565eba49e7cb8190" Nov 03 03:14:38.684600 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.684318 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6996ee59b386b56462e63ecb8f0545da95506e59f9b4cf95565eba49e7cb8190"} err="failed to get container status \"6996ee59b386b56462e63ecb8f0545da95506e59f9b4cf95565eba49e7cb8190\": rpc error: code = NotFound desc = could not find container \"6996ee59b386b56462e63ecb8f0545da95506e59f9b4cf95565eba49e7cb8190\": container with ID starting with 6996ee59b386b56462e63ecb8f0545da95506e59f9b4cf95565eba49e7cb8190 not found: ID does not exist" Nov 03 03:14:38.684600 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.684340 5463 scope.go:122] "RemoveContainer" containerID="8458c21e0225ece4dfd00c018dbe6aeac775d6d6970438db5b4c3de8c05df006" Nov 03 03:14:38.693804 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.693669 5463 scope.go:122] "RemoveContainer" containerID="8458c21e0225ece4dfd00c018dbe6aeac775d6d6970438db5b4c3de8c05df006" Nov 03 03:14:38.694004 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:14:38.693975 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8458c21e0225ece4dfd00c018dbe6aeac775d6d6970438db5b4c3de8c05df006\": container with ID starting with 8458c21e0225ece4dfd00c018dbe6aeac775d6d6970438db5b4c3de8c05df006 not found: ID does not exist" containerID="8458c21e0225ece4dfd00c018dbe6aeac775d6d6970438db5b4c3de8c05df006" Nov 03 03:14:38.694109 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.694011 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8458c21e0225ece4dfd00c018dbe6aeac775d6d6970438db5b4c3de8c05df006"} err="failed to get container status \"8458c21e0225ece4dfd00c018dbe6aeac775d6d6970438db5b4c3de8c05df006\": rpc error: code = NotFound desc = could not find container \"8458c21e0225ece4dfd00c018dbe6aeac775d6d6970438db5b4c3de8c05df006\": container with ID starting with 8458c21e0225ece4dfd00c018dbe6aeac775d6d6970438db5b4c3de8c05df006 not found: ID does not exist" Nov 03 03:14:38.764279 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.764244 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/462caa1d-f10d-4355-ac46-cf9ee0153d23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462caa1d-f10d-4355-ac46-cf9ee0153d23-plugins-config-volume\") pod \"462caa1d-f10d-4355-ac46-cf9ee0153d23\" (UID: \"462caa1d-f10d-4355-ac46-cf9ee0153d23\") " Nov 03 03:14:38.764468 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.764401 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/462caa1d-f10d-4355-ac46-cf9ee0153d23-kube-api-access-t2d8c\" (UniqueName: \"kubernetes.io/projected/462caa1d-f10d-4355-ac46-cf9ee0153d23-kube-api-access-t2d8c\") pod \"462caa1d-f10d-4355-ac46-cf9ee0153d23\" (UID: \"462caa1d-f10d-4355-ac46-cf9ee0153d23\") " Nov 03 03:14:38.764594 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.764571 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/462caa1d-f10d-4355-ac46-cf9ee0153d23-plugins-config-volume" pod "462caa1d-f10d-4355-ac46-cf9ee0153d23" (UID: "462caa1d-f10d-4355-ac46-cf9ee0153d23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:14:38.764740 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.764722 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhsfw\" (UniqueName: \"kubernetes.io/projected/6c5f16c6-739d-4bc0-af05-4587087cc0be-kube-api-access-lhsfw\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:38.764802 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.764743 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c5f16c6-739d-4bc0-af05-4587087cc0be-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:38.764802 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.764756 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462caa1d-f10d-4355-ac46-cf9ee0153d23-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:38.766480 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.766456 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462caa1d-f10d-4355-ac46-cf9ee0153d23-kube-api-access-t2d8c" pod "462caa1d-f10d-4355-ac46-cf9ee0153d23" (UID: "462caa1d-f10d-4355-ac46-cf9ee0153d23"). InnerVolumeSpecName "kube-api-access-t2d8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:14:38.865876 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.865806 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2d8c\" (UniqueName: \"kubernetes.io/projected/462caa1d-f10d-4355-ac46-cf9ee0153d23-kube-api-access-t2d8c\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:38.891346 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.891313 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c"] Nov 03 03:14:38.894584 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:38.894562 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-521f1f-79d8f4c77-msd7c"] Nov 03 03:14:39.098271 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:39.098181 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn"] Nov 03 03:14:39.101602 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:39.101582 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-521f1f-67bcf785b7-56pfn"] Nov 03 03:14:39.801324 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:39.801283 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" containerID="cri-o://8304743d475d541f0121f8b2e7f18e9f228f904cbf9363239a1792deeee2383a" gracePeriod=45 Nov 03 03:14:40.062082 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.062031 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 03:14:40.177796 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.177763 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc1cc20f-fb60-4334-ba14-5f21203ee975-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1cc20f-fb60-4334-ba14-5f21203ee975-tls-certs\") pod \"dc1cc20f-fb60-4334-ba14-5f21203ee975\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " Nov 03 03:14:40.177998 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.177813 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1cc20f-fb60-4334-ba14-5f21203ee975-kube-api-access-r8gm2\" (UniqueName: \"kubernetes.io/projected/dc1cc20f-fb60-4334-ba14-5f21203ee975-kube-api-access-r8gm2\") pod \"dc1cc20f-fb60-4334-ba14-5f21203ee975\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " Nov 03 03:14:40.177998 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.177841 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-model-cache\") pod \"dc1cc20f-fb60-4334-ba14-5f21203ee975\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " Nov 03 03:14:40.177998 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.177887 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-home\" (UniqueName: \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-home\") pod \"dc1cc20f-fb60-4334-ba14-5f21203ee975\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " Nov 03 03:14:40.178182 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.178030 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-tmp-dir\") pod \"dc1cc20f-fb60-4334-ba14-5f21203ee975\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " Nov 03 03:14:40.178182 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.178118 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-model-cache" pod "dc1cc20f-fb60-4334-ba14-5f21203ee975" (UID: "dc1cc20f-fb60-4334-ba14-5f21203ee975"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:14:40.178182 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.178153 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-dshm\") pod \"dc1cc20f-fb60-4334-ba14-5f21203ee975\" (UID: \"dc1cc20f-fb60-4334-ba14-5f21203ee975\") " Nov 03 03:14:40.178377 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.178153 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-home" pod "dc1cc20f-fb60-4334-ba14-5f21203ee975" (UID: "dc1cc20f-fb60-4334-ba14-5f21203ee975"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:14:40.178377 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.178320 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-tmp-dir" pod "dc1cc20f-fb60-4334-ba14-5f21203ee975" (UID: "dc1cc20f-fb60-4334-ba14-5f21203ee975"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:14:40.178576 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.178556 5463 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-model-cache\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:40.178632 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.178580 5463 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-home\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:40.178632 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.178590 5463 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-tmp-dir\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:40.179967 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.179949 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc1cc20f-fb60-4334-ba14-5f21203ee975-tls-certs" pod "dc1cc20f-fb60-4334-ba14-5f21203ee975" (UID: "dc1cc20f-fb60-4334-ba14-5f21203ee975"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:14:40.180394 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.180377 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-dshm" pod "dc1cc20f-fb60-4334-ba14-5f21203ee975" (UID: "dc1cc20f-fb60-4334-ba14-5f21203ee975"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:14:40.180485 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.180406 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1cc20f-fb60-4334-ba14-5f21203ee975-kube-api-access-r8gm2" pod "dc1cc20f-fb60-4334-ba14-5f21203ee975" (UID: "dc1cc20f-fb60-4334-ba14-5f21203ee975"). InnerVolumeSpecName "kube-api-access-r8gm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:14:40.280159 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.280123 5463 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1cc20f-fb60-4334-ba14-5f21203ee975-dshm\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:40.280159 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.280163 5463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1cc20f-fb60-4334-ba14-5f21203ee975-tls-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:40.280407 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.280180 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8gm2\" (UniqueName: \"kubernetes.io/projected/dc1cc20f-fb60-4334-ba14-5f21203ee975-kube-api-access-r8gm2\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:40.441255 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.441227 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="462caa1d-f10d-4355-ac46-cf9ee0153d23" path="/var/lib/kubelet/pods/462caa1d-f10d-4355-ac46-cf9ee0153d23/volumes" Nov 03 03:14:40.441625 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.441613 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c5f16c6-739d-4bc0-af05-4587087cc0be" path="/var/lib/kubelet/pods/6c5f16c6-739d-4bc0-af05-4587087cc0be/volumes" Nov 03 03:14:40.689513 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.689474 5463 generic.go:356] "Generic (PLEG): container finished" podID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerID="8304743d475d541f0121f8b2e7f18e9f228f904cbf9363239a1792deeee2383a" exitCode=0 Nov 03 03:14:40.689513 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.689511 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" Nov 03 03:14:40.689770 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.689545 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" event={"ID":"dc1cc20f-fb60-4334-ba14-5f21203ee975","Type":"ContainerDied","Data":"8304743d475d541f0121f8b2e7f18e9f228f904cbf9363239a1792deeee2383a"} Nov 03 03:14:40.689770 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.689577 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k" event={"ID":"dc1cc20f-fb60-4334-ba14-5f21203ee975","Type":"ContainerDied","Data":"cbe1958a9ceac1e4d122feba58dca9dbf8549227a2752910d721d30a8b1a1d49"} Nov 03 03:14:40.689770 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.689593 5463 scope.go:122] "RemoveContainer" containerID="8304743d475d541f0121f8b2e7f18e9f228f904cbf9363239a1792deeee2383a" Nov 03 03:14:40.698731 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.698711 5463 scope.go:122] "RemoveContainer" containerID="8304743d475d541f0121f8b2e7f18e9f228f904cbf9363239a1792deeee2383a" Nov 03 03:14:40.699037 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:14:40.699017 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8304743d475d541f0121f8b2e7f18e9f228f904cbf9363239a1792deeee2383a\": container with ID starting with 8304743d475d541f0121f8b2e7f18e9f228f904cbf9363239a1792deeee2383a not found: ID does not exist" containerID="8304743d475d541f0121f8b2e7f18e9f228f904cbf9363239a1792deeee2383a" Nov 03 03:14:40.699114 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.699043 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8304743d475d541f0121f8b2e7f18e9f228f904cbf9363239a1792deeee2383a"} err="failed to get container status \"8304743d475d541f0121f8b2e7f18e9f228f904cbf9363239a1792deeee2383a\": rpc error: code = NotFound desc = could not find container \"8304743d475d541f0121f8b2e7f18e9f228f904cbf9363239a1792deeee2383a\": container with ID starting with 8304743d475d541f0121f8b2e7f18e9f228f904cbf9363239a1792deeee2383a not found: ID does not exist" Nov 03 03:14:40.854376 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.854346 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k"] Nov 03 03:14:40.904725 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:40.904693 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/distinct-w2-521f1f-kserve-6f4b9b7756-m6r8k"] Nov 03 03:14:42.440817 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:42.440777 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" path="/var/lib/kubelet/pods/dc1cc20f-fb60-4334-ba14-5f21203ee975/volumes" Nov 03 03:14:43.006877 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:14:43.006839 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:14:44.130008 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:44.129975 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:45.762405 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:45.762371 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:49.652038 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:49.652005 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:49.816442 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:14:49.816386 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:01.579092 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:01.579055 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:01.725579 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:01.725542 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:02.853544 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:02.853508 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:06.086038 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:06.086002 5463 scope.go:122] "RemoveContainer" containerID="8bd750106a039418285bc56dc3afab85825bfe159e6f9bff475868b02db29ced" Nov 03 03:15:06.536201 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:06.536164 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:07.090379 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:07.090342 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:08.112044 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:08.112016 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c4ff9d6d7-pr8r7_f0f91465-9537-4ffb-b2c9-3d74c20ddced/authorino/0.log" Nov 03 03:15:08.486649 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:08.486612 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hvn6t_978d7bd0-c834-455c-90fc-8388993542d6/manager/0.log" Nov 03 03:15:08.602890 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:08.602862 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:08.721617 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:08.721591 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-qczcb_53d43c23-ca28-4ff5-874c-48e53b872e0c/manager/0.log" Nov 03 03:15:08.754428 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:08.754359 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:09.829866 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:09.829823 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-sgplt_a759ef00-49b6-406f-8a47-255bd11ca523/manager/0.log" Nov 03 03:15:10.435225 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:10.435180 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-h8jgm_2d19ed0a-2c09-4b4c-a4bd-cb1f64032fc5/manager/0.log" Nov 03 03:15:11.526052 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:11.526022 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-74bd599f44-rqdlf_9262bb6d-129c-442b-9658-408a64ed30a8/manager/0.log" Nov 03 03:15:11.986264 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:11.986237 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kb5pn_2f2d8c2b-cc81-4620-b32c-a0a056576e35/manager/0.log" Nov 03 03:15:13.007707 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:15:13.007674 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:15:19.898315 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:19.898281 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:20.620717 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:20.620674 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:20.801689 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:20.801661 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:23.771305 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:23.771274 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:23.969632 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:23.969585 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:24.148243 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:24.148207 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:27.008401 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:27.008370 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:27.167082 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:27.167051 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:29.972234 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.972039 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk"] Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973561 5463 cpu_manager.go:414] "containerMap: removing container" podUID="e01f46d8-b6be-4f51-9291-a67a09771b4e" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973590 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e01f46d8-b6be-4f51-9291-a67a09771b4e" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973639 5463 cpu_manager.go:414] "containerMap: removing container" podUID="5ddcf4ff-6e4f-45cd-81a8-8b283429291a" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973648 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ddcf4ff-6e4f-45cd-81a8-8b283429291a" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973660 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a26968f3-5698-4639-aa81-bedc87536bd6" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973666 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a26968f3-5698-4639-aa81-bedc87536bd6" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973686 5463 cpu_manager.go:414] "containerMap: removing container" podUID="462caa1d-f10d-4355-ac46-cf9ee0153d23" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973693 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="462caa1d-f10d-4355-ac46-cf9ee0153d23" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973707 5463 cpu_manager.go:414] "containerMap: removing container" podUID="32df4d57-7e5b-4136-9e75-5c4e9290fd30" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973714 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32df4d57-7e5b-4136-9e75-5c4e9290fd30" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973732 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d683621e-ee05-4efe-9d28-49d1f085d135" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973740 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d683621e-ee05-4efe-9d28-49d1f085d135" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973760 5463 cpu_manager.go:414] "containerMap: removing container" podUID="8f8b0d5f-94b6-439a-bbb0-35fb828131dc" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973768 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8b0d5f-94b6-439a-bbb0-35fb828131dc" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973783 5463 cpu_manager.go:414] "containerMap: removing container" podUID="fcd015dd-30d1-4602-9c58-e7f12e5193d7" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973797 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcd015dd-30d1-4602-9c58-e7f12e5193d7" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973808 5463 cpu_manager.go:414] "containerMap: removing container" podUID="e48e80cd-9263-4d82-8028-2680b548201b" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973815 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e48e80cd-9263-4d82-8028-2680b548201b" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973836 5463 cpu_manager.go:414] "containerMap: removing container" podUID="8f8b0d5f-94b6-439a-bbb0-35fb828131dc" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973843 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8b0d5f-94b6-439a-bbb0-35fb828131dc" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973854 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d6da7ddb-bf04-456f-9821-dfb9d517e0b6" containerName="maas-api" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973860 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6da7ddb-bf04-456f-9821-dfb9d517e0b6" containerName="maas-api" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973879 5463 cpu_manager.go:414] "containerMap: removing container" podUID="9b7b703f-6c31-4881-8e71-a23b27c3e6e7" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973886 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b7b703f-6c31-4881-8e71-a23b27c3e6e7" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973896 5463 cpu_manager.go:414] "containerMap: removing container" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973903 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973918 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a26968f3-5698-4639-aa81-bedc87536bd6" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973925 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a26968f3-5698-4639-aa81-bedc87536bd6" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973951 5463 cpu_manager.go:414] "containerMap: removing container" podUID="53646ed7-2ee1-4a28-9242-033c5be233e4" containerName="main" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973967 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53646ed7-2ee1-4a28-9242-033c5be233e4" containerName="main" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973977 5463 cpu_manager.go:414] "containerMap: removing container" podUID="edb46670-67e5-4fc2-a94d-87634bc91bc4" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.973984 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb46670-67e5-4fc2-a94d-87634bc91bc4" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974003 5463 cpu_manager.go:414] "containerMap: removing container" podUID="ed76fc49-8f0a-4a64-a273-01cb7a2d612b" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974011 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed76fc49-8f0a-4a64-a273-01cb7a2d612b" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974032 5463 cpu_manager.go:414] "containerMap: removing container" podUID="160c3088-0ac6-4694-80b9-67537f5be9eb" containerName="payload-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974040 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160c3088-0ac6-4694-80b9-67537f5be9eb" containerName="payload-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974049 5463 cpu_manager.go:414] "containerMap: removing container" podUID="da6e7a0f-9fd7-4115-9c62-caed63f180de" containerName="payload-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974057 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da6e7a0f-9fd7-4115-9c62-caed63f180de" containerName="payload-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974073 5463 cpu_manager.go:414] "containerMap: removing container" podUID="1ee1477e-0bf0-428b-ac3f-03e5f217116f" containerName="main" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974079 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ee1477e-0bf0-428b-ac3f-03e5f217116f" containerName="main" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974089 5463 cpu_manager.go:414] "containerMap: removing container" podUID="96657503-568c-4ac5-a39c-8abd8eadd843" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974102 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96657503-568c-4ac5-a39c-8abd8eadd843" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974125 5463 cpu_manager.go:414] "containerMap: removing container" podUID="88462923-311f-4302-9503-b1dda3e244e2" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974132 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88462923-311f-4302-9503-b1dda3e244e2" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974140 5463 cpu_manager.go:414] "containerMap: removing container" podUID="258c199f-3518-42ac-8962-d58bc543bc67" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974147 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="258c199f-3518-42ac-8962-d58bc543bc67" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974165 5463 cpu_manager.go:414] "containerMap: removing container" podUID="fcc529d1-9757-467a-bde9-c0afa4822017" containerName="payload-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974171 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc529d1-9757-467a-bde9-c0afa4822017" containerName="payload-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974182 5463 cpu_manager.go:414] "containerMap: removing container" podUID="96657503-568c-4ac5-a39c-8abd8eadd843" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974203 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96657503-568c-4ac5-a39c-8abd8eadd843" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974220 5463 cpu_manager.go:414] "containerMap: removing container" podUID="6c5f16c6-739d-4bc0-af05-4587087cc0be" containerName="payload-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974227 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c5f16c6-739d-4bc0-af05-4587087cc0be" containerName="payload-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974243 5463 cpu_manager.go:414] "containerMap: removing container" podUID="ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974249 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974260 5463 cpu_manager.go:414] "containerMap: removing container" podUID="1ba01dba-9d25-480d-b5d1-f51295ed81b7" containerName="main" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974267 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba01dba-9d25-480d-b5d1-f51295ed81b7" containerName="main" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974287 5463 cpu_manager.go:414] "containerMap: removing container" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974294 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974316 5463 cpu_manager.go:414] "containerMap: removing container" podUID="88462923-311f-4302-9503-b1dda3e244e2" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974323 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88462923-311f-4302-9503-b1dda3e244e2" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974333 5463 cpu_manager.go:414] "containerMap: removing container" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974344 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974359 5463 cpu_manager.go:414] "containerMap: removing container" podUID="699e3934-8850-4132-9f5c-3e446be27484" containerName="payload-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974366 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="699e3934-8850-4132-9f5c-3e446be27484" containerName="payload-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974556 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="88462923-311f-4302-9503-b1dda3e244e2" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974571 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="258c199f-3518-42ac-8962-d58bc543bc67" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974589 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc529d1-9757-467a-bde9-c0afa4822017" containerName="payload-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974605 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="96657503-568c-4ac5-a39c-8abd8eadd843" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974615 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c5f16c6-739d-4bc0-af05-4587087cc0be" containerName="payload-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974631 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ddcf4ff-6e4f-45cd-81a8-8b283429291a" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974640 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae6ff2c5-461a-4dd7-8d8c-22f5f5a4264d" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974654 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba01dba-9d25-480d-b5d1-f51295ed81b7" containerName="main" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974665 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="230d658f-a760-4d30-92f4-b021eda43859" containerName="main" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974682 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="88462923-311f-4302-9503-b1dda3e244e2" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974690 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="544033d3-9c0c-4e56-864c-54fcf9d69d7e" containerName="main" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974703 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="699e3934-8850-4132-9f5c-3e446be27484" containerName="payload-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974722 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e01f46d8-b6be-4f51-9291-a67a09771b4e" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974733 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ddcf4ff-6e4f-45cd-81a8-8b283429291a" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974749 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a26968f3-5698-4639-aa81-bedc87536bd6" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974757 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="462caa1d-f10d-4355-ac46-cf9ee0153d23" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974773 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="32df4d57-7e5b-4136-9e75-5c4e9290fd30" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974783 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d683621e-ee05-4efe-9d28-49d1f085d135" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974799 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f8b0d5f-94b6-439a-bbb0-35fb828131dc" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974815 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcd015dd-30d1-4602-9c58-e7f12e5193d7" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974825 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="e48e80cd-9263-4d82-8028-2680b548201b" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974842 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f8b0d5f-94b6-439a-bbb0-35fb828131dc" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974852 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6da7ddb-bf04-456f-9821-dfb9d517e0b6" containerName="maas-api" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974867 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b7b703f-6c31-4881-8e71-a23b27c3e6e7" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974882 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc1cc20f-fb60-4334-ba14-5f21203ee975" containerName="main" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974891 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a26968f3-5698-4639-aa81-bedc87536bd6" containerName="istio-proxy" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974909 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="53646ed7-2ee1-4a28-9242-033c5be233e4" containerName="main" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974917 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="edb46670-67e5-4fc2-a94d-87634bc91bc4" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974932 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed76fc49-8f0a-4a64-a273-01cb7a2d612b" containerName="payload-pre-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974944 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="160c3088-0ac6-4694-80b9-67537f5be9eb" containerName="payload-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974960 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="da6e7a0f-9fd7-4115-9c62-caed63f180de" containerName="payload-processing" Nov 03 03:15:29.976847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.974970 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ee1477e-0bf0-428b-ac3f-03e5f217116f" containerName="main" Nov 03 03:15:29.981557 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:29.980975 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" Nov 03 03:15:30.031743 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.031716 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk"] Nov 03 03:15:30.080381 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.080356 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qv59l\"" Nov 03 03:15:30.080517 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.080383 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 03:15:30.080517 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.080388 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:15:30.080517 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.080417 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:15:30.080517 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.080425 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0d474c52-serving-cert\"" Nov 03 03:15:30.137465 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.137433 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np9cw\" (UniqueName: \"kubernetes.io/projected/d17ba0c3-3d01-4245-9164-702f9415b0f1-kube-api-access-np9cw\") pod \"maas-api-e2e-ait-0d474c52-744df45b9f-k58pk\" (UID: \"d17ba0c3-3d01-4245-9164-702f9415b0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" Nov 03 03:15:30.137599 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.137496 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17ba0c3-3d01-4245-9164-702f9415b0f1-openshift-service-ca\") pod \"maas-api-e2e-ait-0d474c52-744df45b9f-k58pk\" (UID: \"d17ba0c3-3d01-4245-9164-702f9415b0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" Nov 03 03:15:30.137651 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.137593 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d17ba0c3-3d01-4245-9164-702f9415b0f1-metrics-certs\") pod \"maas-api-e2e-ait-0d474c52-744df45b9f-k58pk\" (UID: \"d17ba0c3-3d01-4245-9164-702f9415b0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" Nov 03 03:15:30.137651 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.137644 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d17ba0c3-3d01-4245-9164-702f9415b0f1-maas-api-tls\") pod \"maas-api-e2e-ait-0d474c52-744df45b9f-k58pk\" (UID: \"d17ba0c3-3d01-4245-9164-702f9415b0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" Nov 03 03:15:30.183247 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.183222 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:30.238436 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.238365 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d17ba0c3-3d01-4245-9164-702f9415b0f1-maas-api-tls\") pod \"maas-api-e2e-ait-0d474c52-744df45b9f-k58pk\" (UID: \"d17ba0c3-3d01-4245-9164-702f9415b0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" Nov 03 03:15:30.238436 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.238410 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np9cw\" (UniqueName: \"kubernetes.io/projected/d17ba0c3-3d01-4245-9164-702f9415b0f1-kube-api-access-np9cw\") pod \"maas-api-e2e-ait-0d474c52-744df45b9f-k58pk\" (UID: \"d17ba0c3-3d01-4245-9164-702f9415b0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" Nov 03 03:15:30.238633 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.238457 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17ba0c3-3d01-4245-9164-702f9415b0f1-openshift-service-ca\") pod \"maas-api-e2e-ait-0d474c52-744df45b9f-k58pk\" (UID: \"d17ba0c3-3d01-4245-9164-702f9415b0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" Nov 03 03:15:30.238633 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.238494 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d17ba0c3-3d01-4245-9164-702f9415b0f1-metrics-certs\") pod \"maas-api-e2e-ait-0d474c52-744df45b9f-k58pk\" (UID: \"d17ba0c3-3d01-4245-9164-702f9415b0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" Nov 03 03:15:30.238633 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:15:30.238516 5463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-serving-cert: secret "maas-api-e2e-ait-0d474c52-serving-cert" not found Nov 03 03:15:30.238633 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:15:30.238600 5463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d17ba0c3-3d01-4245-9164-702f9415b0f1-maas-api-tls podName:d17ba0c3-3d01-4245-9164-702f9415b0f1 nodeName:}" failed. No retries permitted until 2026-11-03 03:15:30.738578932 +0000 UTC m=+5012.422688076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d17ba0c3-3d01-4245-9164-702f9415b0f1-maas-api-tls") pod "maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" (UID: "d17ba0c3-3d01-4245-9164-702f9415b0f1") : secret "maas-api-e2e-ait-0d474c52-serving-cert" not found Nov 03 03:15:30.239118 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.239098 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17ba0c3-3d01-4245-9164-702f9415b0f1-openshift-service-ca\") pod \"maas-api-e2e-ait-0d474c52-744df45b9f-k58pk\" (UID: \"d17ba0c3-3d01-4245-9164-702f9415b0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" Nov 03 03:15:30.240868 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.240846 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d17ba0c3-3d01-4245-9164-702f9415b0f1-metrics-certs\") pod \"maas-api-e2e-ait-0d474c52-744df45b9f-k58pk\" (UID: \"d17ba0c3-3d01-4245-9164-702f9415b0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" Nov 03 03:15:30.298054 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.298023 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np9cw\" (UniqueName: \"kubernetes.io/projected/d17ba0c3-3d01-4245-9164-702f9415b0f1-kube-api-access-np9cw\") pod \"maas-api-e2e-ait-0d474c52-744df45b9f-k58pk\" (UID: \"d17ba0c3-3d01-4245-9164-702f9415b0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" Nov 03 03:15:30.339515 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.339487 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:30.483956 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.483919 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:30.640138 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.640023 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:30.741595 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.741568 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d17ba0c3-3d01-4245-9164-702f9415b0f1-maas-api-tls\") pod \"maas-api-e2e-ait-0d474c52-744df45b9f-k58pk\" (UID: \"d17ba0c3-3d01-4245-9164-702f9415b0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" Nov 03 03:15:30.744037 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.744013 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d17ba0c3-3d01-4245-9164-702f9415b0f1-maas-api-tls\") pod \"maas-api-e2e-ait-0d474c52-744df45b9f-k58pk\" (UID: \"d17ba0c3-3d01-4245-9164-702f9415b0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" Nov 03 03:15:30.894433 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:30.894408 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" Nov 03 03:15:31.274309 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:15:31.274268 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17ba0c3_3d01_4245_9164_702f9415b0f1.slice/crio-265d3b74e65835f04b45c7d39396279503607c8c3d4519c53949ebf99a713f1f WatchSource:0}: Error finding container 265d3b74e65835f04b45c7d39396279503607c8c3d4519c53949ebf99a713f1f: Status 404 returned error can't find the container with id 265d3b74e65835f04b45c7d39396279503607c8c3d4519c53949ebf99a713f1f Nov 03 03:15:31.275593 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:31.275571 5463 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:15:31.277802 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:31.277775 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk"] Nov 03 03:15:31.961513 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:31.961481 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" event={"ID":"d17ba0c3-3d01-4245-9164-702f9415b0f1","Type":"ContainerStarted","Data":"265d3b74e65835f04b45c7d39396279503607c8c3d4519c53949ebf99a713f1f"} Nov 03 03:15:32.968739 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:32.968696 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" event={"ID":"d17ba0c3-3d01-4245-9164-702f9415b0f1","Type":"ContainerStarted","Data":"9971356f2aee013d590821cb15720a0eaa14397753ab1c0bc5ddb2eb74eab8fe"} Nov 03 03:15:32.969210 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:32.968869 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" Nov 03 03:15:33.086896 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:33.086836 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" podStartSLOduration=3.057399322 podStartE2EDuration="4.086823627s" totalImagesPullingTime="1.029424305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:33.085790204 +0000 UTC m=+5014.769899350" watchObservedRunningTime="2026-11-03 03:15:33.086823627 +0000 UTC m=+5014.770932781" Nov 03 03:15:35.059797 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:35.059765 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:38.980655 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:38.980624 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" Nov 03 03:15:39.849307 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:39.849273 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:43.006874 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:15:43.006838 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:15:44.622538 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:44.622505 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:49.759604 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:49.759564 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:54.937737 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:54.937704 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:59.737536 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:15:59.737496 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:16:02.664836 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:02.664800 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:16:02.828489 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:02.828446 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:16:04.525447 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:04.525414 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:16:08.591023 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:08.590993 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk"] Nov 03 03:16:08.591606 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:08.591174 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" podUID="d17ba0c3-3d01-4245-9164-702f9415b0f1" containerName="maas-api" containerID="cri-o://9971356f2aee013d590821cb15720a0eaa14397753ab1c0bc5ddb2eb74eab8fe" gracePeriod=30 Nov 03 03:16:08.724812 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:08.724782 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:16:08.847608 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:08.847547 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" Nov 03 03:16:08.982094 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:08.982058 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d17ba0c3-3d01-4245-9164-702f9415b0f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17ba0c3-3d01-4245-9164-702f9415b0f1-openshift-service-ca\") pod \"d17ba0c3-3d01-4245-9164-702f9415b0f1\" (UID: \"d17ba0c3-3d01-4245-9164-702f9415b0f1\") " Nov 03 03:16:08.982319 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:08.982141 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d17ba0c3-3d01-4245-9164-702f9415b0f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d17ba0c3-3d01-4245-9164-702f9415b0f1-maas-api-tls\") pod \"d17ba0c3-3d01-4245-9164-702f9415b0f1\" (UID: \"d17ba0c3-3d01-4245-9164-702f9415b0f1\") " Nov 03 03:16:08.982319 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:08.982290 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d17ba0c3-3d01-4245-9164-702f9415b0f1-kube-api-access-np9cw\" (UniqueName: \"kubernetes.io/projected/d17ba0c3-3d01-4245-9164-702f9415b0f1-kube-api-access-np9cw\") pod \"d17ba0c3-3d01-4245-9164-702f9415b0f1\" (UID: \"d17ba0c3-3d01-4245-9164-702f9415b0f1\") " Nov 03 03:16:08.982469 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:08.982359 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d17ba0c3-3d01-4245-9164-702f9415b0f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d17ba0c3-3d01-4245-9164-702f9415b0f1-metrics-certs\") pod \"d17ba0c3-3d01-4245-9164-702f9415b0f1\" (UID: \"d17ba0c3-3d01-4245-9164-702f9415b0f1\") " Nov 03 03:16:08.982469 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:08.982446 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d17ba0c3-3d01-4245-9164-702f9415b0f1-openshift-service-ca" pod "d17ba0c3-3d01-4245-9164-702f9415b0f1" (UID: "d17ba0c3-3d01-4245-9164-702f9415b0f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:16:08.982675 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:08.982653 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17ba0c3-3d01-4245-9164-702f9415b0f1-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:08.984390 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:08.984363 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17ba0c3-3d01-4245-9164-702f9415b0f1-kube-api-access-np9cw" pod "d17ba0c3-3d01-4245-9164-702f9415b0f1" (UID: "d17ba0c3-3d01-4245-9164-702f9415b0f1"). InnerVolumeSpecName "kube-api-access-np9cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:08.984518 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:08.984426 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d17ba0c3-3d01-4245-9164-702f9415b0f1-metrics-certs" pod "d17ba0c3-3d01-4245-9164-702f9415b0f1" (UID: "d17ba0c3-3d01-4245-9164-702f9415b0f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:16:08.984518 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:08.984456 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d17ba0c3-3d01-4245-9164-702f9415b0f1-maas-api-tls" pod "d17ba0c3-3d01-4245-9164-702f9415b0f1" (UID: "d17ba0c3-3d01-4245-9164-702f9415b0f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:16:09.083510 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:09.083438 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np9cw\" (UniqueName: \"kubernetes.io/projected/d17ba0c3-3d01-4245-9164-702f9415b0f1-kube-api-access-np9cw\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:09.083510 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:09.083469 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d17ba0c3-3d01-4245-9164-702f9415b0f1-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:09.083510 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:09.083479 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d17ba0c3-3d01-4245-9164-702f9415b0f1-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:16:09.151901 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:09.151867 5463 generic.go:356] "Generic (PLEG): container finished" podID="d17ba0c3-3d01-4245-9164-702f9415b0f1" containerID="9971356f2aee013d590821cb15720a0eaa14397753ab1c0bc5ddb2eb74eab8fe" exitCode=0 Nov 03 03:16:09.152065 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:09.151915 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" Nov 03 03:16:09.152065 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:09.151950 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" event={"ID":"d17ba0c3-3d01-4245-9164-702f9415b0f1","Type":"ContainerDied","Data":"9971356f2aee013d590821cb15720a0eaa14397753ab1c0bc5ddb2eb74eab8fe"} Nov 03 03:16:09.152065 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:09.151996 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk" event={"ID":"d17ba0c3-3d01-4245-9164-702f9415b0f1","Type":"ContainerDied","Data":"265d3b74e65835f04b45c7d39396279503607c8c3d4519c53949ebf99a713f1f"} Nov 03 03:16:09.152065 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:09.152015 5463 scope.go:122] "RemoveContainer" containerID="9971356f2aee013d590821cb15720a0eaa14397753ab1c0bc5ddb2eb74eab8fe" Nov 03 03:16:09.162272 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:09.162251 5463 scope.go:122] "RemoveContainer" containerID="9971356f2aee013d590821cb15720a0eaa14397753ab1c0bc5ddb2eb74eab8fe" Nov 03 03:16:09.162554 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:16:09.162531 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9971356f2aee013d590821cb15720a0eaa14397753ab1c0bc5ddb2eb74eab8fe\": container with ID starting with 9971356f2aee013d590821cb15720a0eaa14397753ab1c0bc5ddb2eb74eab8fe not found: ID does not exist" containerID="9971356f2aee013d590821cb15720a0eaa14397753ab1c0bc5ddb2eb74eab8fe" Nov 03 03:16:09.162663 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:09.162559 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9971356f2aee013d590821cb15720a0eaa14397753ab1c0bc5ddb2eb74eab8fe"} err="failed to get container status \"9971356f2aee013d590821cb15720a0eaa14397753ab1c0bc5ddb2eb74eab8fe\": rpc error: code = NotFound desc = could not find container \"9971356f2aee013d590821cb15720a0eaa14397753ab1c0bc5ddb2eb74eab8fe\": container with ID starting with 9971356f2aee013d590821cb15720a0eaa14397753ab1c0bc5ddb2eb74eab8fe not found: ID does not exist" Nov 03 03:16:09.316596 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:09.316557 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk"] Nov 03 03:16:09.322534 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:09.322508 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d474c52-744df45b9f-k58pk"] Nov 03 03:16:10.264155 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:10.264118 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:16:10.441127 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:10.441091 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d17ba0c3-3d01-4245-9164-702f9415b0f1" path="/var/lib/kubelet/pods/d17ba0c3-3d01-4245-9164-702f9415b0f1/volumes" Nov 03 03:16:10.500746 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:10.500717 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:16:11.998051 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:11.998016 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:16:13.007256 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:16:13.007221 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:16:30.763454 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:30.763420 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:16:43.005645 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:16:43.005610 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:16:53.241719 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:53.241674 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:16:56.386180 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:56.386141 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:16:58.618382 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:16:58.618332 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:17:02.355770 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.355732 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm"] Nov 03 03:17:02.356401 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.356379 5463 cpu_manager.go:414] "containerMap: removing container" podUID="5ddcf4ff-6e4f-45cd-81a8-8b283429291a" containerName="istio-proxy" Nov 03 03:17:02.356491 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.356403 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ddcf4ff-6e4f-45cd-81a8-8b283429291a" containerName="istio-proxy" Nov 03 03:17:02.356491 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.356463 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d17ba0c3-3d01-4245-9164-702f9415b0f1" containerName="maas-api" Nov 03 03:17:02.356491 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.356473 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d17ba0c3-3d01-4245-9164-702f9415b0f1" containerName="maas-api" Nov 03 03:17:02.356656 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.356573 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="96657503-568c-4ac5-a39c-8abd8eadd843" containerName="istio-proxy" Nov 03 03:17:02.356656 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.356589 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d17ba0c3-3d01-4245-9164-702f9415b0f1" containerName="maas-api" Nov 03 03:17:02.359841 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.359821 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.406288 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.406264 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8"] Nov 03 03:17:02.409597 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.409582 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.456886 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.456854 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm"] Nov 03 03:17:02.469456 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.469427 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd738f58-993a-4da6-b860-9e79f58184f0-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.469608 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.469462 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd738f58-993a-4da6-b860-9e79f58184f0-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.469608 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.469496 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd738f58-993a-4da6-b860-9e79f58184f0-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.469748 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.469611 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd738f58-993a-4da6-b860-9e79f58184f0-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.469748 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.469699 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd738f58-993a-4da6-b860-9e79f58184f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.469834 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.469747 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd738f58-993a-4da6-b860-9e79f58184f0-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.469834 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.469802 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ncsb\" (UniqueName: \"kubernetes.io/projected/fd738f58-993a-4da6-b860-9e79f58184f0-kube-api-access-8ncsb\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.469934 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.469839 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd738f58-993a-4da6-b860-9e79f58184f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.469934 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.469856 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd738f58-993a-4da6-b860-9e79f58184f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.504411 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.504385 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8"] Nov 03 03:17:02.570889 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.570853 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecff7992-20d8-4f1b-8266-243263d93d7a-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.570889 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.570889 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecff7992-20d8-4f1b-8266-243263d93d7a-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.571096 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.570907 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecff7992-20d8-4f1b-8266-243263d93d7a-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.571096 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.570988 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd738f58-993a-4da6-b860-9e79f58184f0-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.571173 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.571114 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd738f58-993a-4da6-b860-9e79f58184f0-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.571243 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.571182 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd738f58-993a-4da6-b860-9e79f58184f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.571306 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.571241 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecff7992-20d8-4f1b-8266-243263d93d7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.571371 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.571310 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd738f58-993a-4da6-b860-9e79f58184f0-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.571371 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.571335 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ncsb\" (UniqueName: \"kubernetes.io/projected/fd738f58-993a-4da6-b860-9e79f58184f0-kube-api-access-8ncsb\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.571494 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.571387 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd738f58-993a-4da6-b860-9e79f58184f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.571494 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.571419 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbh6t\" (UniqueName: \"kubernetes.io/projected/ecff7992-20d8-4f1b-8266-243263d93d7a-kube-api-access-xbh6t\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.571494 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.571447 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd738f58-993a-4da6-b860-9e79f58184f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.571494 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.571478 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd738f58-993a-4da6-b860-9e79f58184f0-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.571494 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.571492 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecff7992-20d8-4f1b-8266-243263d93d7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.571777 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.571525 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd738f58-993a-4da6-b860-9e79f58184f0-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.571777 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.571605 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd738f58-993a-4da6-b860-9e79f58184f0-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.571777 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.571725 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd738f58-993a-4da6-b860-9e79f58184f0-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.571959 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.571775 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd738f58-993a-4da6-b860-9e79f58184f0-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.571959 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.571817 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecff7992-20d8-4f1b-8266-243263d93d7a-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.571959 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.571850 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecff7992-20d8-4f1b-8266-243263d93d7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.571959 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.571876 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecff7992-20d8-4f1b-8266-243263d93d7a-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.572138 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.571973 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd738f58-993a-4da6-b860-9e79f58184f0-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.572138 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.572011 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd738f58-993a-4da6-b860-9e79f58184f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.573810 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.573787 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd738f58-993a-4da6-b860-9e79f58184f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.574081 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.574059 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd738f58-993a-4da6-b860-9e79f58184f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.629495 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.629436 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd738f58-993a-4da6-b860-9e79f58184f0-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.629603 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.629578 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ncsb\" (UniqueName: \"kubernetes.io/projected/fd738f58-993a-4da6-b860-9e79f58184f0-kube-api-access-8ncsb\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bwphm\" (UID: \"fd738f58-993a-4da6-b860-9e79f58184f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.672679 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.672649 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecff7992-20d8-4f1b-8266-243263d93d7a-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.672824 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.672685 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecff7992-20d8-4f1b-8266-243263d93d7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.672824 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.672702 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecff7992-20d8-4f1b-8266-243263d93d7a-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.672824 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.672721 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecff7992-20d8-4f1b-8266-243263d93d7a-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.672824 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.672735 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecff7992-20d8-4f1b-8266-243263d93d7a-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.672824 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.672752 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecff7992-20d8-4f1b-8266-243263d93d7a-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.672824 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.672767 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:02.672824 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.672792 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecff7992-20d8-4f1b-8266-243263d93d7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.673227 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.672857 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbh6t\" (UniqueName: \"kubernetes.io/projected/ecff7992-20d8-4f1b-8266-243263d93d7a-kube-api-access-xbh6t\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.673227 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.672902 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecff7992-20d8-4f1b-8266-243263d93d7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.673227 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.673099 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecff7992-20d8-4f1b-8266-243263d93d7a-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.673227 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.673177 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecff7992-20d8-4f1b-8266-243263d93d7a-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.673458 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.673310 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecff7992-20d8-4f1b-8266-243263d93d7a-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.673458 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.673369 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecff7992-20d8-4f1b-8266-243263d93d7a-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.673649 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.673628 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecff7992-20d8-4f1b-8266-243263d93d7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.674909 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.674883 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecff7992-20d8-4f1b-8266-243263d93d7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.675328 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.675306 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecff7992-20d8-4f1b-8266-243263d93d7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.732318 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.730637 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbh6t\" (UniqueName: \"kubernetes.io/projected/ecff7992-20d8-4f1b-8266-243263d93d7a-kube-api-access-xbh6t\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:02.732318 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:02.730957 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecff7992-20d8-4f1b-8266-243263d93d7a-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sl8s8\" (UID: \"ecff7992-20d8-4f1b-8266-243263d93d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:03.021971 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:03.021930 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:03.087080 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:17:03.087046 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd738f58_993a_4da6_b860_9e79f58184f0.slice/crio-3faee726a3ef090a4b5768bb397da571033ea17a514fbcc521fad02d38ab684f WatchSource:0}: Error finding container 3faee726a3ef090a4b5768bb397da571033ea17a514fbcc521fad02d38ab684f: Status 404 returned error can't find the container with id 3faee726a3ef090a4b5768bb397da571033ea17a514fbcc521fad02d38ab684f Nov 03 03:17:03.089340 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:03.089305 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:17:03.089454 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:03.089388 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:17:03.089454 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:03.089433 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:17:03.090000 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:03.089980 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm"] Nov 03 03:17:03.358479 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:03.358405 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 03:17:03.367747 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:03.367719 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 03:17:03.392701 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:17:03.392668 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecff7992_20d8_4f1b_8266_243263d93d7a.slice/crio-ccf70b5f63ee3b24e87c0f41b9ffb559a282ffc2facc331b3d82ac214ac46d4d WatchSource:0}: Error finding container ccf70b5f63ee3b24e87c0f41b9ffb559a282ffc2facc331b3d82ac214ac46d4d: Status 404 returned error can't find the container with id ccf70b5f63ee3b24e87c0f41b9ffb559a282ffc2facc331b3d82ac214ac46d4d Nov 03 03:17:03.393804 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:03.393785 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8"] Nov 03 03:17:03.394947 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:03.394904 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:17:03.395027 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:03.394999 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:17:03.395075 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:03.395045 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:17:03.426986 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:03.426954 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" event={"ID":"fd738f58-993a-4da6-b860-9e79f58184f0","Type":"ContainerStarted","Data":"2773e56e10c99f8c3273187e2e183d2a17da1e609394eac205c7d81ec1faefd8"} Nov 03 03:17:03.427097 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:03.427021 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" event={"ID":"fd738f58-993a-4da6-b860-9e79f58184f0","Type":"ContainerStarted","Data":"3faee726a3ef090a4b5768bb397da571033ea17a514fbcc521fad02d38ab684f"} Nov 03 03:17:03.428448 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:03.428424 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" event={"ID":"ecff7992-20d8-4f1b-8266-243263d93d7a","Type":"ContainerStarted","Data":"ccf70b5f63ee3b24e87c0f41b9ffb559a282ffc2facc331b3d82ac214ac46d4d"} Nov 03 03:17:03.547272 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:03.547213 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" podStartSLOduration=1.547174619 podStartE2EDuration="1.547174619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:17:03.54637295 +0000 UTC m=+5105.230482104" watchObservedRunningTime="2026-11-03 03:17:03.547174619 +0000 UTC m=+5105.231283773" Nov 03 03:17:03.679363 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:03.679320 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:04.440433 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:04.440400 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" event={"ID":"ecff7992-20d8-4f1b-8266-243263d93d7a","Type":"ContainerStarted","Data":"d96c4c1c791e175b44bc75e3c3ec32f917cee56885e160c65c0ad9d32459c8ce"} Nov 03 03:17:04.680661 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:04.680620 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bwphm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Nov 03 03:17:04.680964 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:04.680937 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" podUID="fd738f58-993a-4da6-b860-9e79f58184f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Nov 03 03:17:05.022505 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:05.022476 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:05.023834 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:05.023808 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-sl8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Nov 03 03:17:05.023915 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:05.023865 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" podUID="ecff7992-20d8-4f1b-8266-243263d93d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Nov 03 03:17:05.678213 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:05.677572 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bwphm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Nov 03 03:17:05.678213 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:05.677651 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" podUID="fd738f58-993a-4da6-b860-9e79f58184f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Nov 03 03:17:06.022361 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:06.022275 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-sl8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Nov 03 03:17:06.022361 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:06.022347 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" podUID="ecff7992-20d8-4f1b-8266-243263d93d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Nov 03 03:17:06.680658 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:06.680614 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bwphm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Nov 03 03:17:06.681302 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:06.681267 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" podUID="fd738f58-993a-4da6-b860-9e79f58184f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Nov 03 03:17:07.023109 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:07.023024 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-sl8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Nov 03 03:17:07.023109 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:07.023081 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" podUID="ecff7992-20d8-4f1b-8266-243263d93d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Nov 03 03:17:07.682800 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:07.682764 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bwphm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Nov 03 03:17:07.683552 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:07.683517 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" podUID="fd738f58-993a-4da6-b860-9e79f58184f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Nov 03 03:17:08.022482 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:08.022396 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-sl8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Nov 03 03:17:08.022712 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:08.022689 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" podUID="ecff7992-20d8-4f1b-8266-243263d93d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Nov 03 03:17:08.674618 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:08.674575 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bwphm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Nov 03 03:17:08.674934 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:08.674905 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" podUID="fd738f58-993a-4da6-b860-9e79f58184f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Nov 03 03:17:09.023629 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:09.023536 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-sl8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Nov 03 03:17:09.024170 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:09.024140 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" podUID="ecff7992-20d8-4f1b-8266-243263d93d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Nov 03 03:17:09.673633 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:09.673595 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bwphm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Nov 03 03:17:09.673823 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:09.673663 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" podUID="fd738f58-993a-4da6-b860-9e79f58184f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Nov 03 03:17:10.022472 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:10.022389 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-sl8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Nov 03 03:17:10.022472 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:10.022444 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" podUID="ecff7992-20d8-4f1b-8266-243263d93d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Nov 03 03:17:10.673959 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:10.673916 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bwphm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Nov 03 03:17:10.674405 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:10.673979 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" podUID="fd738f58-993a-4da6-b860-9e79f58184f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Nov 03 03:17:11.682123 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:11.682089 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:11.682635 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:11.682559 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:11.683414 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:11.683391 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" Nov 03 03:17:11.802272 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:11.802243 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:11.802620 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:11.802595 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:11.803378 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:11.803358 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" Nov 03 03:17:11.900570 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:11.900519 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sl8s8" podStartSLOduration=9.900505853 podStartE2EDuration="9.900505853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:17:04.671678825 +0000 UTC m=+5106.355787981" watchObservedRunningTime="2026-11-03 03:17:11.900505853 +0000 UTC m=+5113.584615008" Nov 03 03:17:13.006661 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:17:13.006622 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:17:31.142577 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:17:31.142538 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:17:43.005824 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:17:43.005745 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:18:03.784207 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:03.784155 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2"] Nov 03 03:18:03.787708 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:03.787690 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2" Nov 03 03:18:03.840465 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:03.840430 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2"] Nov 03 03:18:03.888417 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:03.888384 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:18:03.888417 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:03.888397 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:18:03.888611 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:03.888489 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:18:03.888611 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:03.888490 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:18:03.961400 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:03.961370 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-f6gw2\" (UID: \"5c454515-5f30-4bdc-ad12-e0e2ff6bb51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2" Nov 03 03:18:03.961575 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:03.961412 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwvrq\" (UniqueName: \"kubernetes.io/projected/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-kube-api-access-kwvrq\") pod \"maas-api-revoke-sub-f03629db-f6gw2\" (UID: \"5c454515-5f30-4bdc-ad12-e0e2ff6bb51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2" Nov 03 03:18:03.961575 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:03.961489 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-f6gw2\" (UID: \"5c454515-5f30-4bdc-ad12-e0e2ff6bb51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2" Nov 03 03:18:04.062601 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:04.062525 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-f6gw2\" (UID: \"5c454515-5f30-4bdc-ad12-e0e2ff6bb51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2" Nov 03 03:18:04.062765 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:04.062613 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-f6gw2\" (UID: \"5c454515-5f30-4bdc-ad12-e0e2ff6bb51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2" Nov 03 03:18:04.062765 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:04.062642 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwvrq\" (UniqueName: \"kubernetes.io/projected/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-kube-api-access-kwvrq\") pod \"maas-api-revoke-sub-f03629db-f6gw2\" (UID: \"5c454515-5f30-4bdc-ad12-e0e2ff6bb51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2" Nov 03 03:18:04.063282 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:04.063261 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-f6gw2\" (UID: \"5c454515-5f30-4bdc-ad12-e0e2ff6bb51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2" Nov 03 03:18:04.064828 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:04.064804 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-f6gw2\" (UID: \"5c454515-5f30-4bdc-ad12-e0e2ff6bb51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2" Nov 03 03:18:04.120345 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:04.120311 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwvrq\" (UniqueName: \"kubernetes.io/projected/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-kube-api-access-kwvrq\") pod \"maas-api-revoke-sub-f03629db-f6gw2\" (UID: \"5c454515-5f30-4bdc-ad12-e0e2ff6bb51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2" Nov 03 03:18:04.400592 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:04.400566 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2" Nov 03 03:18:04.770864 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:18:04.770821 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c454515_5f30_4bdc_ad12_e0e2ff6bb51f.slice/crio-4ee2c854c5d7228a24770f3aef0b78550d870b07def3d9b742ed937e4a39ecae WatchSource:0}: Error finding container 4ee2c854c5d7228a24770f3aef0b78550d870b07def3d9b742ed937e4a39ecae: Status 404 returned error can't find the container with id 4ee2c854c5d7228a24770f3aef0b78550d870b07def3d9b742ed937e4a39ecae Nov 03 03:18:04.773469 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:04.773437 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2"] Nov 03 03:18:04.830255 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:04.830224 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2" event={"ID":"5c454515-5f30-4bdc-ad12-e0e2ff6bb51f","Type":"ContainerStarted","Data":"4ee2c854c5d7228a24770f3aef0b78550d870b07def3d9b742ed937e4a39ecae"} Nov 03 03:18:05.836845 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:05.836810 5463 generic.go:356] "Generic (PLEG): container finished" podID="5c454515-5f30-4bdc-ad12-e0e2ff6bb51f" containerID="22f87b55d92d7b333c9918210e730a3a0c80fc563ac42661e546cc470534f3bd" exitCode=0 Nov 03 03:18:05.837249 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:05.836886 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2" event={"ID":"5c454515-5f30-4bdc-ad12-e0e2ff6bb51f","Type":"ContainerDied","Data":"22f87b55d92d7b333c9918210e730a3a0c80fc563ac42661e546cc470534f3bd"} Nov 03 03:18:06.978620 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:06.978591 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2" Nov 03 03:18:06.989982 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:06.989950 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-maas-api-openshift-ca\") pod \"5c454515-5f30-4bdc-ad12-e0e2ff6bb51f\" (UID: \"5c454515-5f30-4bdc-ad12-e0e2ff6bb51f\") " Nov 03 03:18:06.990129 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:06.990010 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-kube-api-access-kwvrq\" (UniqueName: \"kubernetes.io/projected/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-kube-api-access-kwvrq\") pod \"5c454515-5f30-4bdc-ad12-e0e2ff6bb51f\" (UID: \"5c454515-5f30-4bdc-ad12-e0e2ff6bb51f\") " Nov 03 03:18:06.990129 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:06.990042 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-maas-api-xks-ca\") pod \"5c454515-5f30-4bdc-ad12-e0e2ff6bb51f\" (UID: \"5c454515-5f30-4bdc-ad12-e0e2ff6bb51f\") " Nov 03 03:18:06.990359 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:06.990336 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-maas-api-openshift-ca" pod "5c454515-5f30-4bdc-ad12-e0e2ff6bb51f" (UID: "5c454515-5f30-4bdc-ad12-e0e2ff6bb51f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:18:06.992170 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:06.992142 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-maas-api-xks-ca" pod "5c454515-5f30-4bdc-ad12-e0e2ff6bb51f" (UID: "5c454515-5f30-4bdc-ad12-e0e2ff6bb51f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:18:06.992339 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:06.992173 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-kube-api-access-kwvrq" pod "5c454515-5f30-4bdc-ad12-e0e2ff6bb51f" (UID: "5c454515-5f30-4bdc-ad12-e0e2ff6bb51f"). InnerVolumeSpecName "kube-api-access-kwvrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:18:07.091695 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:07.091664 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:18:07.091695 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:07.091695 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwvrq\" (UniqueName: \"kubernetes.io/projected/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-kube-api-access-kwvrq\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:18:07.091964 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:07.091710 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:18:07.851394 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:07.851358 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2" event={"ID":"5c454515-5f30-4bdc-ad12-e0e2ff6bb51f","Type":"ContainerDied","Data":"4ee2c854c5d7228a24770f3aef0b78550d870b07def3d9b742ed937e4a39ecae"} Nov 03 03:18:07.851394 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:07.851375 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2" Nov 03 03:18:07.851671 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:07.851410 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ee2c854c5d7228a24770f3aef0b78550d870b07def3d9b742ed937e4a39ecae" Nov 03 03:18:13.007311 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:18:13.007272 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:18:17.286138 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:17.286098 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:18:23.453856 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:23.453823 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq"] Nov 03 03:18:23.454369 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:23.454350 5463 cpu_manager.go:414] "containerMap: removing container" podUID="5c454515-5f30-4bdc-ad12-e0e2ff6bb51f" containerName="revoke-keys" Nov 03 03:18:23.454369 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:23.454369 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c454515-5f30-4bdc-ad12-e0e2ff6bb51f" containerName="revoke-keys" Nov 03 03:18:23.454475 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:23.454466 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c454515-5f30-4bdc-ad12-e0e2ff6bb51f" containerName="revoke-keys" Nov 03 03:18:23.457463 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:23.457445 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq" Nov 03 03:18:23.508921 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:23.508891 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq"] Nov 03 03:18:23.544889 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:23.544860 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3189d1cf-2221-4253-949d-4a976ca07ec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-d9lpq\" (UID: \"3189d1cf-2221-4253-949d-4a976ca07ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq" Nov 03 03:18:23.545010 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:23.544897 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3189d1cf-2221-4253-949d-4a976ca07ec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-d9lpq\" (UID: \"3189d1cf-2221-4253-949d-4a976ca07ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq" Nov 03 03:18:23.545010 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:23.544999 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ddg9\" (UniqueName: \"kubernetes.io/projected/3189d1cf-2221-4253-949d-4a976ca07ec3-kube-api-access-7ddg9\") pod \"maas-api-revoke-sub-0165d7d2-d9lpq\" (UID: \"3189d1cf-2221-4253-949d-4a976ca07ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq" Nov 03 03:18:23.557438 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:23.557414 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:18:23.557584 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:23.557448 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:18:23.557584 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:23.557481 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:18:23.560278 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:23.560254 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:18:23.646107 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:23.646073 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ddg9\" (UniqueName: \"kubernetes.io/projected/3189d1cf-2221-4253-949d-4a976ca07ec3-kube-api-access-7ddg9\") pod \"maas-api-revoke-sub-0165d7d2-d9lpq\" (UID: \"3189d1cf-2221-4253-949d-4a976ca07ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq" Nov 03 03:18:23.646297 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:23.646223 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3189d1cf-2221-4253-949d-4a976ca07ec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-d9lpq\" (UID: \"3189d1cf-2221-4253-949d-4a976ca07ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq" Nov 03 03:18:23.646297 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:23.646265 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3189d1cf-2221-4253-949d-4a976ca07ec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-d9lpq\" (UID: \"3189d1cf-2221-4253-949d-4a976ca07ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq" Nov 03 03:18:23.646964 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:23.646941 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3189d1cf-2221-4253-949d-4a976ca07ec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-d9lpq\" (UID: \"3189d1cf-2221-4253-949d-4a976ca07ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq" Nov 03 03:18:23.648716 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:23.648690 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3189d1cf-2221-4253-949d-4a976ca07ec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-d9lpq\" (UID: \"3189d1cf-2221-4253-949d-4a976ca07ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq" Nov 03 03:18:23.703470 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:23.703443 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ddg9\" (UniqueName: \"kubernetes.io/projected/3189d1cf-2221-4253-949d-4a976ca07ec3-kube-api-access-7ddg9\") pod \"maas-api-revoke-sub-0165d7d2-d9lpq\" (UID: \"3189d1cf-2221-4253-949d-4a976ca07ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq" Nov 03 03:18:23.770302 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:23.770249 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq" Nov 03 03:18:24.135245 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:18:24.135213 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3189d1cf_2221_4253_949d_4a976ca07ec3.slice/crio-2b81ec0f0576c12741a65b887135f45eeb469782dfcc8e315343cccdf625b129 WatchSource:0}: Error finding container 2b81ec0f0576c12741a65b887135f45eeb469782dfcc8e315343cccdf625b129: Status 404 returned error can't find the container with id 2b81ec0f0576c12741a65b887135f45eeb469782dfcc8e315343cccdf625b129 Nov 03 03:18:24.136626 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:24.136603 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq"] Nov 03 03:18:24.957108 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:24.957072 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq" event={"ID":"3189d1cf-2221-4253-949d-4a976ca07ec3","Type":"ContainerStarted","Data":"7347f46639e287331054ef663b12b31fccc8f29b665a1cff59f8fb7718c31ba2"} Nov 03 03:18:24.957533 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:24.957133 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq" event={"ID":"3189d1cf-2221-4253-949d-4a976ca07ec3","Type":"ContainerStarted","Data":"2b81ec0f0576c12741a65b887135f45eeb469782dfcc8e315343cccdf625b129"} Nov 03 03:18:25.071768 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:25.071720 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq" podStartSLOduration=2.071706503 podStartE2EDuration="2.071706503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:18:25.069847796 +0000 UTC m=+5186.753956951" watchObservedRunningTime="2026-11-03 03:18:25.071706503 +0000 UTC m=+5186.755815657" Nov 03 03:18:25.963777 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:25.963744 5463 generic.go:356] "Generic (PLEG): container finished" podID="3189d1cf-2221-4253-949d-4a976ca07ec3" containerID="7347f46639e287331054ef663b12b31fccc8f29b665a1cff59f8fb7718c31ba2" exitCode=0 Nov 03 03:18:25.964214 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:25.963785 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq" event={"ID":"3189d1cf-2221-4253-949d-4a976ca07ec3","Type":"ContainerDied","Data":"7347f46639e287331054ef663b12b31fccc8f29b665a1cff59f8fb7718c31ba2"} Nov 03 03:18:26.480289 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:26.480254 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:18:26.621575 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:26.621542 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:18:27.103451 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:27.103424 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq" Nov 03 03:18:27.182471 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:27.182443 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3189d1cf-2221-4253-949d-4a976ca07ec3-kube-api-access-7ddg9\" (UniqueName: \"kubernetes.io/projected/3189d1cf-2221-4253-949d-4a976ca07ec3-kube-api-access-7ddg9\") pod \"3189d1cf-2221-4253-949d-4a976ca07ec3\" (UID: \"3189d1cf-2221-4253-949d-4a976ca07ec3\") " Nov 03 03:18:27.182601 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:27.182501 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3189d1cf-2221-4253-949d-4a976ca07ec3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3189d1cf-2221-4253-949d-4a976ca07ec3-maas-api-xks-ca\") pod \"3189d1cf-2221-4253-949d-4a976ca07ec3\" (UID: \"3189d1cf-2221-4253-949d-4a976ca07ec3\") " Nov 03 03:18:27.182601 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:27.182557 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3189d1cf-2221-4253-949d-4a976ca07ec3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3189d1cf-2221-4253-949d-4a976ca07ec3-maas-api-openshift-ca\") pod \"3189d1cf-2221-4253-949d-4a976ca07ec3\" (UID: \"3189d1cf-2221-4253-949d-4a976ca07ec3\") " Nov 03 03:18:27.182924 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:27.182903 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3189d1cf-2221-4253-949d-4a976ca07ec3-maas-api-openshift-ca" pod "3189d1cf-2221-4253-949d-4a976ca07ec3" (UID: "3189d1cf-2221-4253-949d-4a976ca07ec3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:18:27.184568 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:27.184546 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3189d1cf-2221-4253-949d-4a976ca07ec3-maas-api-xks-ca" pod "3189d1cf-2221-4253-949d-4a976ca07ec3" (UID: "3189d1cf-2221-4253-949d-4a976ca07ec3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:18:27.184700 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:27.184681 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3189d1cf-2221-4253-949d-4a976ca07ec3-kube-api-access-7ddg9" pod "3189d1cf-2221-4253-949d-4a976ca07ec3" (UID: "3189d1cf-2221-4253-949d-4a976ca07ec3"). InnerVolumeSpecName "kube-api-access-7ddg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:18:27.283652 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:27.283583 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3189d1cf-2221-4253-949d-4a976ca07ec3-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:18:27.283652 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:27.283610 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3189d1cf-2221-4253-949d-4a976ca07ec3-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:18:27.283652 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:27.283625 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ddg9\" (UniqueName: \"kubernetes.io/projected/3189d1cf-2221-4253-949d-4a976ca07ec3-kube-api-access-7ddg9\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:18:27.975216 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:27.975171 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq" Nov 03 03:18:27.975426 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:27.975223 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq" event={"ID":"3189d1cf-2221-4253-949d-4a976ca07ec3","Type":"ContainerDied","Data":"2b81ec0f0576c12741a65b887135f45eeb469782dfcc8e315343cccdf625b129"} Nov 03 03:18:27.975426 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:27.975265 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b81ec0f0576c12741a65b887135f45eeb469782dfcc8e315343cccdf625b129" Nov 03 03:18:31.398544 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:31.398511 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:18:35.252725 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.252696 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4"] Nov 03 03:18:35.253234 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.253130 5463 cpu_manager.go:414] "containerMap: removing container" podUID="3189d1cf-2221-4253-949d-4a976ca07ec3" containerName="revoke-keys" Nov 03 03:18:35.253234 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.253141 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3189d1cf-2221-4253-949d-4a976ca07ec3" containerName="revoke-keys" Nov 03 03:18:35.253234 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.253233 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="3189d1cf-2221-4253-949d-4a976ca07ec3" containerName="revoke-keys" Nov 03 03:18:35.256295 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.256271 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4" Nov 03 03:18:35.309225 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.309180 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4"] Nov 03 03:18:35.342236 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.342204 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:18:35.342397 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.342330 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:18:35.342481 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.342465 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:18:35.356899 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.356877 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:18:35.357232 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.357210 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-mz5n4\" (UID: \"cbe62b8f-fc1b-4bcf-99cd-280030dff7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4" Nov 03 03:18:35.357333 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.357255 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-mz5n4\" (UID: \"cbe62b8f-fc1b-4bcf-99cd-280030dff7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4" Nov 03 03:18:35.357333 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.357272 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6vfb\" (UniqueName: \"kubernetes.io/projected/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-kube-api-access-l6vfb\") pod \"maas-api-revoke-sub-6b19338f-mz5n4\" (UID: \"cbe62b8f-fc1b-4bcf-99cd-280030dff7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4" Nov 03 03:18:35.458635 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.458592 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-mz5n4\" (UID: \"cbe62b8f-fc1b-4bcf-99cd-280030dff7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4" Nov 03 03:18:35.458797 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.458656 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-mz5n4\" (UID: \"cbe62b8f-fc1b-4bcf-99cd-280030dff7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4" Nov 03 03:18:35.458797 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.458682 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6vfb\" (UniqueName: \"kubernetes.io/projected/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-kube-api-access-l6vfb\") pod \"maas-api-revoke-sub-6b19338f-mz5n4\" (UID: \"cbe62b8f-fc1b-4bcf-99cd-280030dff7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4" Nov 03 03:18:35.459312 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.459291 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-mz5n4\" (UID: \"cbe62b8f-fc1b-4bcf-99cd-280030dff7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4" Nov 03 03:18:35.460885 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.460863 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-mz5n4\" (UID: \"cbe62b8f-fc1b-4bcf-99cd-280030dff7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4" Nov 03 03:18:35.517451 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.517383 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6vfb\" (UniqueName: \"kubernetes.io/projected/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-kube-api-access-l6vfb\") pod \"maas-api-revoke-sub-6b19338f-mz5n4\" (UID: \"cbe62b8f-fc1b-4bcf-99cd-280030dff7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4" Nov 03 03:18:35.572315 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.572290 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4" Nov 03 03:18:35.940809 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:18:35.940766 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe62b8f_fc1b_4bcf_99cd_280030dff7d6.slice/crio-dbe19c97a00a7dae543d797214f613039c369afa24000b63574c7d9dbfff3fa7 WatchSource:0}: Error finding container dbe19c97a00a7dae543d797214f613039c369afa24000b63574c7d9dbfff3fa7: Status 404 returned error can't find the container with id dbe19c97a00a7dae543d797214f613039c369afa24000b63574c7d9dbfff3fa7 Nov 03 03:18:35.941628 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:35.941599 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4"] Nov 03 03:18:36.019044 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:36.019012 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4" event={"ID":"cbe62b8f-fc1b-4bcf-99cd-280030dff7d6","Type":"ContainerStarted","Data":"dbe19c97a00a7dae543d797214f613039c369afa24000b63574c7d9dbfff3fa7"} Nov 03 03:18:37.025757 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:37.025724 5463 generic.go:356] "Generic (PLEG): container finished" podID="cbe62b8f-fc1b-4bcf-99cd-280030dff7d6" containerID="e172dd191d03c64376ece83d4f53f65785ea02d113f9145f4a2203ee2dedfd92" exitCode=0 Nov 03 03:18:37.026221 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:37.025816 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4" event={"ID":"cbe62b8f-fc1b-4bcf-99cd-280030dff7d6","Type":"ContainerDied","Data":"e172dd191d03c64376ece83d4f53f65785ea02d113f9145f4a2203ee2dedfd92"} Nov 03 03:18:38.167362 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:38.167338 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4" Nov 03 03:18:38.289307 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:38.289226 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-maas-api-openshift-ca\") pod \"cbe62b8f-fc1b-4bcf-99cd-280030dff7d6\" (UID: \"cbe62b8f-fc1b-4bcf-99cd-280030dff7d6\") " Nov 03 03:18:38.289307 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:38.289287 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-maas-api-xks-ca\") pod \"cbe62b8f-fc1b-4bcf-99cd-280030dff7d6\" (UID: \"cbe62b8f-fc1b-4bcf-99cd-280030dff7d6\") " Nov 03 03:18:38.289544 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:38.289412 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-kube-api-access-l6vfb\" (UniqueName: \"kubernetes.io/projected/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-kube-api-access-l6vfb\") pod \"cbe62b8f-fc1b-4bcf-99cd-280030dff7d6\" (UID: \"cbe62b8f-fc1b-4bcf-99cd-280030dff7d6\") " Nov 03 03:18:38.289618 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:38.289536 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-maas-api-openshift-ca" pod "cbe62b8f-fc1b-4bcf-99cd-280030dff7d6" (UID: "cbe62b8f-fc1b-4bcf-99cd-280030dff7d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:18:38.289803 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:38.289778 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:18:38.291447 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:38.291423 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-maas-api-xks-ca" pod "cbe62b8f-fc1b-4bcf-99cd-280030dff7d6" (UID: "cbe62b8f-fc1b-4bcf-99cd-280030dff7d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:18:38.291534 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:38.291510 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-kube-api-access-l6vfb" pod "cbe62b8f-fc1b-4bcf-99cd-280030dff7d6" (UID: "cbe62b8f-fc1b-4bcf-99cd-280030dff7d6"). InnerVolumeSpecName "kube-api-access-l6vfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:18:38.390967 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:38.390933 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:18:38.390967 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:38.390963 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6vfb\" (UniqueName: \"kubernetes.io/projected/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6-kube-api-access-l6vfb\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:18:39.037653 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:39.037624 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4" Nov 03 03:18:39.037862 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:39.037663 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4" event={"ID":"cbe62b8f-fc1b-4bcf-99cd-280030dff7d6","Type":"ContainerDied","Data":"dbe19c97a00a7dae543d797214f613039c369afa24000b63574c7d9dbfff3fa7"} Nov 03 03:18:39.037862 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:39.037714 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbe19c97a00a7dae543d797214f613039c369afa24000b63574c7d9dbfff3fa7" Nov 03 03:18:43.005487 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:18:43.005454 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:18:52.003467 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:52.003430 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:18:52.520245 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:52.520182 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7"] Nov 03 03:18:52.520789 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:52.520767 5463 cpu_manager.go:414] "containerMap: removing container" podUID="cbe62b8f-fc1b-4bcf-99cd-280030dff7d6" containerName="revoke-keys" Nov 03 03:18:52.520900 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:52.520790 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe62b8f-fc1b-4bcf-99cd-280030dff7d6" containerName="revoke-keys" Nov 03 03:18:52.520969 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:52.520935 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe62b8f-fc1b-4bcf-99cd-280030dff7d6" containerName="revoke-keys" Nov 03 03:18:52.524361 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:52.524341 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" Nov 03 03:18:52.550627 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:52.550597 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a40b232-bc77-47ca-999c-5ecaca771e50-tls-certs\") pod \"e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7\" (UID: \"5a40b232-bc77-47ca-999c-5ecaca771e50\") " pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" Nov 03 03:18:52.550778 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:52.550652 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgjmf\" (UniqueName: \"kubernetes.io/projected/5a40b232-bc77-47ca-999c-5ecaca771e50-kube-api-access-rgjmf\") pod \"e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7\" (UID: \"5a40b232-bc77-47ca-999c-5ecaca771e50\") " pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" Nov 03 03:18:52.577003 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:52.576970 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7"] Nov 03 03:18:52.624547 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:52.624507 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:18:52.625259 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:52.625237 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-e934465c-epp-sa-dockercfg-7qlw8\"" Nov 03 03:18:52.625399 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:52.625272 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-e934465c-kserve-self-signed-certs\"" Nov 03 03:18:52.651597 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:52.651571 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a40b232-bc77-47ca-999c-5ecaca771e50-tls-certs\") pod \"e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7\" (UID: \"5a40b232-bc77-47ca-999c-5ecaca771e50\") " pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" Nov 03 03:18:52.651733 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:52.651613 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgjmf\" (UniqueName: \"kubernetes.io/projected/5a40b232-bc77-47ca-999c-5ecaca771e50-kube-api-access-rgjmf\") pod \"e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7\" (UID: \"5a40b232-bc77-47ca-999c-5ecaca771e50\") " pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" Nov 03 03:18:52.654959 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:52.654930 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a40b232-bc77-47ca-999c-5ecaca771e50-tls-certs\") pod \"e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7\" (UID: \"5a40b232-bc77-47ca-999c-5ecaca771e50\") " pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" Nov 03 03:18:52.709216 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:52.709163 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgjmf\" (UniqueName: \"kubernetes.io/projected/5a40b232-bc77-47ca-999c-5ecaca771e50-kube-api-access-rgjmf\") pod \"e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7\" (UID: \"5a40b232-bc77-47ca-999c-5ecaca771e50\") " pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" Nov 03 03:18:52.838992 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:52.838911 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" Nov 03 03:18:53.208037 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:18:53.207988 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a40b232_bc77_47ca_999c_5ecaca771e50.slice/crio-780a12678cf3b86510015320fefdc401930c3e194db78810c7fc27eb044879a3 WatchSource:0}: Error finding container 780a12678cf3b86510015320fefdc401930c3e194db78810c7fc27eb044879a3: Status 404 returned error can't find the container with id 780a12678cf3b86510015320fefdc401930c3e194db78810c7fc27eb044879a3 Nov 03 03:18:53.210290 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:53.210265 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7"] Nov 03 03:18:54.150750 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:54.150719 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" event={"ID":"5a40b232-bc77-47ca-999c-5ecaca771e50","Type":"ContainerStarted","Data":"780a12678cf3b86510015320fefdc401930c3e194db78810c7fc27eb044879a3"} Nov 03 03:18:56.164053 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:56.164013 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" event={"ID":"5a40b232-bc77-47ca-999c-5ecaca771e50","Type":"ContainerStarted","Data":"5730445752ab96a3a1c588976d83c3e3657c6309ab96be431c39fe2549fe18e8"} Nov 03 03:18:56.164465 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:56.164087 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" Nov 03 03:18:56.278766 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:18:56.278711 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" podStartSLOduration=2.262583705 podStartE2EDuration="4.278697321s" totalImagesPullingTime="2.016113616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:18:56.278036685 +0000 UTC m=+5217.962145842" watchObservedRunningTime="2026-11-03 03:18:56.278697321 +0000 UTC m=+5217.962806476" Nov 03 03:19:10.334279 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:19:10.334170 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:19:13.006596 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:19:13.006557 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:19:27.171519 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:19:27.171488 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" Nov 03 03:19:31.723488 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:19:31.723442 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:19:42.731342 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:19:42.731301 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bwphm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:19:42.731863 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:19:42.731356 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bwphm" podUID="fd738f58-993a-4da6-b860-9e79f58184f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:19:43.017676 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:19:43.017590 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:20:04.945459 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:04.945420 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7"] Nov 03 03:20:04.979758 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:04.979717 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:05.134476 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:05.134442 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:07.171270 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:07.171232 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" podUID="5a40b232-bc77-47ca-999c-5ecaca771e50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:20:13.043599 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:20:13.043556 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:20:17.170863 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:17.170826 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" podUID="5a40b232-bc77-47ca-999c-5ecaca771e50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:20:19.984873 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:19.984811 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" podUID="5a40b232-bc77-47ca-999c-5ecaca771e50" containerName="main" containerID="cri-o://5730445752ab96a3a1c588976d83c3e3657c6309ab96be431c39fe2549fe18e8" gracePeriod=45 Nov 03 03:20:23.442185 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:23.442134 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:24.042342 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:24.042267 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:24.415362 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:24.415332 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:27.182218 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:27.182091 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" podUID="5a40b232-bc77-47ca-999c-5ecaca771e50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:20:27.182758 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:27.182622 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" Nov 03 03:20:28.441070 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:28.441028 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:28.684216 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:28.683285 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:28.887911 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:28.887823 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:31.683973 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:31.683905 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:31.850785 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:31.850740 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:32.050774 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:32.050692 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:35.146621 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:35.146591 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:35.311490 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:35.311444 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:35.472600 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:35.472496 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:35.631117 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:35.631079 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:36.978399 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:36.978358 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:37.128343 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:37.128312 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:37.171342 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:37.171306 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" podUID="5a40b232-bc77-47ca-999c-5ecaca771e50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:20:40.092167 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:40.092089 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:43.006713 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:20:43.006679 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:20:45.018400 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:45.018366 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:47.171918 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:47.171881 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" podUID="5a40b232-bc77-47ca-999c-5ecaca771e50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:20:49.895059 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:49.895029 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:50.235050 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:50.235022 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" Nov 03 03:20:50.344497 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:50.344469 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a40b232-bc77-47ca-999c-5ecaca771e50-kube-api-access-rgjmf\" (UniqueName: \"kubernetes.io/projected/5a40b232-bc77-47ca-999c-5ecaca771e50-kube-api-access-rgjmf\") pod \"5a40b232-bc77-47ca-999c-5ecaca771e50\" (UID: \"5a40b232-bc77-47ca-999c-5ecaca771e50\") " Nov 03 03:20:50.344672 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:50.344657 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a40b232-bc77-47ca-999c-5ecaca771e50-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a40b232-bc77-47ca-999c-5ecaca771e50-tls-certs\") pod \"5a40b232-bc77-47ca-999c-5ecaca771e50\" (UID: \"5a40b232-bc77-47ca-999c-5ecaca771e50\") " Nov 03 03:20:50.347246 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:50.347183 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a40b232-bc77-47ca-999c-5ecaca771e50-tls-certs" pod "5a40b232-bc77-47ca-999c-5ecaca771e50" (UID: "5a40b232-bc77-47ca-999c-5ecaca771e50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:20:50.348019 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:50.347992 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a40b232-bc77-47ca-999c-5ecaca771e50-kube-api-access-rgjmf" pod "5a40b232-bc77-47ca-999c-5ecaca771e50" (UID: "5a40b232-bc77-47ca-999c-5ecaca771e50"). InnerVolumeSpecName "kube-api-access-rgjmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:20:50.445340 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:50.445274 5463 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a40b232-bc77-47ca-999c-5ecaca771e50-tls-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:20:50.445340 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:50.445297 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgjmf\" (UniqueName: \"kubernetes.io/projected/5a40b232-bc77-47ca-999c-5ecaca771e50-kube-api-access-rgjmf\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:20:50.865911 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:50.865822 5463 generic.go:356] "Generic (PLEG): container finished" podID="5a40b232-bc77-47ca-999c-5ecaca771e50" containerID="5730445752ab96a3a1c588976d83c3e3657c6309ab96be431c39fe2549fe18e8" exitCode=0 Nov 03 03:20:50.865911 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:50.865880 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" Nov 03 03:20:50.866177 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:50.865912 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" event={"ID":"5a40b232-bc77-47ca-999c-5ecaca771e50","Type":"ContainerDied","Data":"5730445752ab96a3a1c588976d83c3e3657c6309ab96be431c39fe2549fe18e8"} Nov 03 03:20:50.866177 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:50.866015 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7" event={"ID":"5a40b232-bc77-47ca-999c-5ecaca771e50","Type":"ContainerDied","Data":"780a12678cf3b86510015320fefdc401930c3e194db78810c7fc27eb044879a3"} Nov 03 03:20:50.866177 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:50.866038 5463 scope.go:122] "RemoveContainer" containerID="5730445752ab96a3a1c588976d83c3e3657c6309ab96be431c39fe2549fe18e8" Nov 03 03:20:50.875246 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:50.875230 5463 scope.go:122] "RemoveContainer" containerID="5730445752ab96a3a1c588976d83c3e3657c6309ab96be431c39fe2549fe18e8" Nov 03 03:20:50.875512 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:20:50.875484 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5730445752ab96a3a1c588976d83c3e3657c6309ab96be431c39fe2549fe18e8\": container with ID starting with 5730445752ab96a3a1c588976d83c3e3657c6309ab96be431c39fe2549fe18e8 not found: ID does not exist" containerID="5730445752ab96a3a1c588976d83c3e3657c6309ab96be431c39fe2549fe18e8" Nov 03 03:20:50.875619 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:50.875517 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5730445752ab96a3a1c588976d83c3e3657c6309ab96be431c39fe2549fe18e8"} err="failed to get container status \"5730445752ab96a3a1c588976d83c3e3657c6309ab96be431c39fe2549fe18e8\": rpc error: code = NotFound desc = could not find container \"5730445752ab96a3a1c588976d83c3e3657c6309ab96be431c39fe2549fe18e8\": container with ID starting with 5730445752ab96a3a1c588976d83c3e3657c6309ab96be431c39fe2549fe18e8 not found: ID does not exist" Nov 03 03:20:51.031180 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:51.031152 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7"] Nov 03 03:20:51.081406 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:51.081377 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-e934465c-kserve-router-scheduler-ff9b4956c-b96z7"] Nov 03 03:20:52.441862 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:52.441833 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a40b232-bc77-47ca-999c-5ecaca771e50" path="/var/lib/kubelet/pods/5a40b232-bc77-47ca-999c-5ecaca771e50/volumes" Nov 03 03:20:53.346579 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:53.346548 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:53.999666 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:53.999632 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:54.150448 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:54.150422 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:54.314945 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:54.314860 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:55.125689 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:55.125650 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:56.629731 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:56.629693 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:20:59.982162 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:20:59.982127 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:21:04.801739 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:04.801694 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:21:09.601479 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:09.601436 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:21:13.006274 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:21:13.006240 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:21:14.398652 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:14.398619 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:21:19.174252 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:19.174207 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:21:23.999891 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:23.999858 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:21:28.371598 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:28.371559 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:21:28.701286 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:28.701251 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:21:32.365145 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:32.365112 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:21:33.493573 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:33.493539 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:21:38.210762 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:38.210723 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:21:42.279370 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.279339 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg"] Nov 03 03:21:42.279780 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.279766 5463 cpu_manager.go:414] "containerMap: removing container" podUID="5a40b232-bc77-47ca-999c-5ecaca771e50" containerName="main" Nov 03 03:21:42.279826 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.279782 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a40b232-bc77-47ca-999c-5ecaca771e50" containerName="main" Nov 03 03:21:42.279863 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.279856 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a40b232-bc77-47ca-999c-5ecaca771e50" containerName="main" Nov 03 03:21:42.283161 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.283144 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg" Nov 03 03:21:42.336280 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.336250 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg"] Nov 03 03:21:42.383412 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.383383 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:21:42.383558 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.383421 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:21:42.383558 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.383430 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:21:42.385859 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.385839 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:21:42.443853 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.443823 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d952ddac-f153-414e-90f8-31e20b03e42c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db0ccd66-hgtxg\" (UID: \"d952ddac-f153-414e-90f8-31e20b03e42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg" Nov 03 03:21:42.444042 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.443884 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzqkt\" (UniqueName: \"kubernetes.io/projected/d952ddac-f153-414e-90f8-31e20b03e42c-kube-api-access-pzqkt\") pod \"maas-api-revoke-sub-db0ccd66-hgtxg\" (UID: \"d952ddac-f153-414e-90f8-31e20b03e42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg" Nov 03 03:21:42.444042 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.443993 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d952ddac-f153-414e-90f8-31e20b03e42c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db0ccd66-hgtxg\" (UID: \"d952ddac-f153-414e-90f8-31e20b03e42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg" Nov 03 03:21:42.545173 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.545089 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d952ddac-f153-414e-90f8-31e20b03e42c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db0ccd66-hgtxg\" (UID: \"d952ddac-f153-414e-90f8-31e20b03e42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg" Nov 03 03:21:42.545173 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.545152 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqkt\" (UniqueName: \"kubernetes.io/projected/d952ddac-f153-414e-90f8-31e20b03e42c-kube-api-access-pzqkt\") pod \"maas-api-revoke-sub-db0ccd66-hgtxg\" (UID: \"d952ddac-f153-414e-90f8-31e20b03e42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg" Nov 03 03:21:42.545173 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.545171 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d952ddac-f153-414e-90f8-31e20b03e42c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db0ccd66-hgtxg\" (UID: \"d952ddac-f153-414e-90f8-31e20b03e42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg" Nov 03 03:21:42.545888 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.545857 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d952ddac-f153-414e-90f8-31e20b03e42c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db0ccd66-hgtxg\" (UID: \"d952ddac-f153-414e-90f8-31e20b03e42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg" Nov 03 03:21:42.547478 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.547450 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d952ddac-f153-414e-90f8-31e20b03e42c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db0ccd66-hgtxg\" (UID: \"d952ddac-f153-414e-90f8-31e20b03e42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg" Nov 03 03:21:42.602138 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.602110 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzqkt\" (UniqueName: \"kubernetes.io/projected/d952ddac-f153-414e-90f8-31e20b03e42c-kube-api-access-pzqkt\") pod \"maas-api-revoke-sub-db0ccd66-hgtxg\" (UID: \"d952ddac-f153-414e-90f8-31e20b03e42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg" Nov 03 03:21:42.895570 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.895539 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg" Nov 03 03:21:42.987660 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:42.987600 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:21:43.013176 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:21:43.013121 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:21:43.265550 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:21:43.265513 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd952ddac_f153_414e_90f8_31e20b03e42c.slice/crio-e7c65f87889b60a5c10450ffb0f4063c3504c64901b5ff0e70bd5894a33fc962 WatchSource:0}: Error finding container e7c65f87889b60a5c10450ffb0f4063c3504c64901b5ff0e70bd5894a33fc962: Status 404 returned error can't find the container with id e7c65f87889b60a5c10450ffb0f4063c3504c64901b5ff0e70bd5894a33fc962 Nov 03 03:21:43.267843 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:43.267822 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg"] Nov 03 03:21:44.156515 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:44.156478 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg" event={"ID":"d952ddac-f153-414e-90f8-31e20b03e42c","Type":"ContainerStarted","Data":"c60f1eea75a0da4cd4982756557c740b610b205f84916a17334848a1676d2250"} Nov 03 03:21:44.156977 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:44.156544 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg" event={"ID":"d952ddac-f153-414e-90f8-31e20b03e42c","Type":"ContainerStarted","Data":"e7c65f87889b60a5c10450ffb0f4063c3504c64901b5ff0e70bd5894a33fc962"} Nov 03 03:21:44.271877 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:44.271823 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg" podStartSLOduration=2.271809754 podStartE2EDuration="2.271809754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:21:44.270336385 +0000 UTC m=+5385.954445543" watchObservedRunningTime="2026-11-03 03:21:44.271809754 +0000 UTC m=+5385.955918903" Nov 03 03:21:45.163834 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:45.163802 5463 generic.go:356] "Generic (PLEG): container finished" podID="d952ddac-f153-414e-90f8-31e20b03e42c" containerID="c60f1eea75a0da4cd4982756557c740b610b205f84916a17334848a1676d2250" exitCode=0 Nov 03 03:21:45.164357 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:45.163952 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg" event={"ID":"d952ddac-f153-414e-90f8-31e20b03e42c","Type":"ContainerDied","Data":"c60f1eea75a0da4cd4982756557c740b610b205f84916a17334848a1676d2250"} Nov 03 03:21:46.299472 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:46.299444 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg" Nov 03 03:21:46.385570 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:46.385542 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d952ddac-f153-414e-90f8-31e20b03e42c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d952ddac-f153-414e-90f8-31e20b03e42c-maas-api-openshift-ca\") pod \"d952ddac-f153-414e-90f8-31e20b03e42c\" (UID: \"d952ddac-f153-414e-90f8-31e20b03e42c\") " Nov 03 03:21:46.385718 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:46.385676 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d952ddac-f153-414e-90f8-31e20b03e42c-kube-api-access-pzqkt\" (UniqueName: \"kubernetes.io/projected/d952ddac-f153-414e-90f8-31e20b03e42c-kube-api-access-pzqkt\") pod \"d952ddac-f153-414e-90f8-31e20b03e42c\" (UID: \"d952ddac-f153-414e-90f8-31e20b03e42c\") " Nov 03 03:21:46.385762 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:46.385743 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d952ddac-f153-414e-90f8-31e20b03e42c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d952ddac-f153-414e-90f8-31e20b03e42c-maas-api-xks-ca\") pod \"d952ddac-f153-414e-90f8-31e20b03e42c\" (UID: \"d952ddac-f153-414e-90f8-31e20b03e42c\") " Nov 03 03:21:46.385908 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:46.385888 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d952ddac-f153-414e-90f8-31e20b03e42c-maas-api-openshift-ca" pod "d952ddac-f153-414e-90f8-31e20b03e42c" (UID: "d952ddac-f153-414e-90f8-31e20b03e42c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:21:46.386057 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:46.386038 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d952ddac-f153-414e-90f8-31e20b03e42c-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:21:46.387762 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:46.387731 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d952ddac-f153-414e-90f8-31e20b03e42c-maas-api-xks-ca" pod "d952ddac-f153-414e-90f8-31e20b03e42c" (UID: "d952ddac-f153-414e-90f8-31e20b03e42c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:21:46.387844 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:46.387768 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d952ddac-f153-414e-90f8-31e20b03e42c-kube-api-access-pzqkt" pod "d952ddac-f153-414e-90f8-31e20b03e42c" (UID: "d952ddac-f153-414e-90f8-31e20b03e42c"). InnerVolumeSpecName "kube-api-access-pzqkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:21:46.486952 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:46.486923 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d952ddac-f153-414e-90f8-31e20b03e42c-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:21:46.486952 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:46.486948 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzqkt\" (UniqueName: \"kubernetes.io/projected/d952ddac-f153-414e-90f8-31e20b03e42c-kube-api-access-pzqkt\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:21:47.175515 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:47.175483 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg" Nov 03 03:21:47.175697 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:47.175510 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg" event={"ID":"d952ddac-f153-414e-90f8-31e20b03e42c","Type":"ContainerDied","Data":"e7c65f87889b60a5c10450ffb0f4063c3504c64901b5ff0e70bd5894a33fc962"} Nov 03 03:21:47.175697 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:47.175563 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7c65f87889b60a5c10450ffb0f4063c3504c64901b5ff0e70bd5894a33fc962" Nov 03 03:21:47.724473 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:47.724443 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:21:52.480746 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:52.480705 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:21:53.438998 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:53.438945 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:21:53.926139 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:53.926090 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:21:57.211398 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:57.211350 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:21:58.843382 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:58.843346 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:21:59.806041 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:21:59.805980 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:22:00.853825 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:00.853784 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:22:01.023144 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:01.023114 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:22:01.532094 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:01.532055 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:22:03.403499 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:03.403470 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 03:22:03.411750 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:03.411718 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 03:22:05.759170 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:05.759135 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:22:05.931578 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:05.931521 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:22:13.005896 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:22:13.005809 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:22:27.594241 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:27.594185 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:22:32.737231 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:32.737176 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:22:34.898935 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:34.898901 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:22:35.367540 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:35.367444 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:22:35.528017 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:35.527974 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:22:36.525908 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:36.525874 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:22:37.425545 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.425511 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w"] Nov 03 03:22:37.426109 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.426088 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d952ddac-f153-414e-90f8-31e20b03e42c" containerName="revoke-keys" Nov 03 03:22:37.426109 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.426108 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d952ddac-f153-414e-90f8-31e20b03e42c" containerName="revoke-keys" Nov 03 03:22:37.426311 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.426260 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d952ddac-f153-414e-90f8-31e20b03e42c" containerName="revoke-keys" Nov 03 03:22:37.429448 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.429429 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" Nov 03 03:22:37.462637 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.462605 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w\" (UID: \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" Nov 03 03:22:37.462784 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.462656 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kqvn\" (UniqueName: \"kubernetes.io/projected/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-kube-api-access-8kqvn\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w\" (UID: \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" Nov 03 03:22:37.462784 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.462759 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w\" (UID: \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" Nov 03 03:22:37.462891 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.462843 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w\" (UID: \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" Nov 03 03:22:37.483118 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.483091 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w"] Nov 03 03:22:37.530999 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.530967 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-2232ad-serving-cert\"" Nov 03 03:22:37.530999 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.530993 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qv59l\"" Nov 03 03:22:37.531454 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.530980 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:22:37.531454 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.530980 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:22:37.532569 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.532551 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 03:22:37.564488 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.564448 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w\" (UID: \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" Nov 03 03:22:37.564691 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.564518 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kqvn\" (UniqueName: \"kubernetes.io/projected/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-kube-api-access-8kqvn\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w\" (UID: \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" Nov 03 03:22:37.564691 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.564554 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w\" (UID: \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" Nov 03 03:22:37.564691 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.564641 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w\" (UID: \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" Nov 03 03:22:37.565499 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:22:37.565475 5463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-serving-cert: secret "maas-api-e2e-worker-single-main-2232ad-serving-cert" not found Nov 03 03:22:37.565755 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:22:37.565731 5463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-maas-api-tls podName:28ad1a21-b4f5-4af8-9c90-fc6ce500774f nodeName:}" failed. No retries permitted until 2026-11-03 03:22:38.06570626 +0000 UTC m=+5439.749815405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-maas-api-tls") pod "maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" (UID: "28ad1a21-b4f5-4af8-9c90-fc6ce500774f") : secret "maas-api-e2e-worker-single-main-2232ad-serving-cert" not found Nov 03 03:22:37.566488 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.566464 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w\" (UID: \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" Nov 03 03:22:37.568032 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.568005 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w\" (UID: \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" Nov 03 03:22:37.622867 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.622838 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kqvn\" (UniqueName: \"kubernetes.io/projected/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-kube-api-access-8kqvn\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w\" (UID: \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" Nov 03 03:22:37.663892 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.663863 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:22:37.897023 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:37.896996 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:22:38.048136 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:38.048102 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:22:38.070871 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:38.070835 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w\" (UID: \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" Nov 03 03:22:38.073310 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:38.073282 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w\" (UID: \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" Nov 03 03:22:38.341998 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:38.341909 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" Nov 03 03:22:38.710661 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:22:38.710621 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ad1a21_b4f5_4af8_9c90_fc6ce500774f.slice/crio-c18ae6d3a5d11b506c8ade85df5193d91dafa78a0eab79aa25445c01b04806df WatchSource:0}: Error finding container c18ae6d3a5d11b506c8ade85df5193d91dafa78a0eab79aa25445c01b04806df: Status 404 returned error can't find the container with id c18ae6d3a5d11b506c8ade85df5193d91dafa78a0eab79aa25445c01b04806df Nov 03 03:22:38.712941 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:38.712917 5463 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:22:38.713076 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:38.713003 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w"] Nov 03 03:22:39.461349 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:39.461317 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" event={"ID":"28ad1a21-b4f5-4af8-9c90-fc6ce500774f","Type":"ContainerStarted","Data":"c18ae6d3a5d11b506c8ade85df5193d91dafa78a0eab79aa25445c01b04806df"} Nov 03 03:22:40.467300 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:40.467239 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" event={"ID":"28ad1a21-b4f5-4af8-9c90-fc6ce500774f","Type":"ContainerStarted","Data":"3adc87ede73663f345a147749eadefe4eb51d24a0ec47e61721538a49818581f"} Nov 03 03:22:40.467887 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:40.467386 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" Nov 03 03:22:40.584540 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:40.584486 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" podStartSLOduration=2.522806185 podStartE2EDuration="3.584468928s" totalImagesPullingTime="1.061662743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:22:40.582876198 +0000 UTC m=+5442.266985354" watchObservedRunningTime="2026-11-03 03:22:40.584468928 +0000 UTC m=+5442.268578101" Nov 03 03:22:42.351623 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:42.351590 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:22:42.509154 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:42.509122 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:22:42.693395 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:42.693365 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:22:43.006916 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:22:43.006836 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:22:46.479530 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:46.479496 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" Nov 03 03:22:47.498940 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:47.498898 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:22:52.189855 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:52.189820 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:22:56.986771 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:22:56.986737 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:23:01.691112 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:01.691077 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:23:05.741186 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:05.741145 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w"] Nov 03 03:23:05.743703 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:05.741412 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" podUID="28ad1a21-b4f5-4af8-9c90-fc6ce500774f" containerName="maas-api" containerID="cri-o://3adc87ede73663f345a147749eadefe4eb51d24a0ec47e61721538a49818581f" gracePeriod=30 Nov 03 03:23:05.862659 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:05.862627 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:23:06.007073 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.007016 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" Nov 03 03:23:06.150123 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.150091 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-maas-api-tls\") pod \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\" (UID: \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\") " Nov 03 03:23:06.150323 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.150171 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-kube-api-access-8kqvn\" (UniqueName: \"kubernetes.io/projected/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-kube-api-access-8kqvn\") pod \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\" (UID: \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\") " Nov 03 03:23:06.150323 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.150247 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-metrics-certs\") pod \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\" (UID: \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\") " Nov 03 03:23:06.150468 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.150362 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-openshift-service-ca\") pod \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\" (UID: \"28ad1a21-b4f5-4af8-9c90-fc6ce500774f\") " Nov 03 03:23:06.150757 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.150729 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-openshift-service-ca" pod "28ad1a21-b4f5-4af8-9c90-fc6ce500774f" (UID: "28ad1a21-b4f5-4af8-9c90-fc6ce500774f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:23:06.152423 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.152396 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-maas-api-tls" pod "28ad1a21-b4f5-4af8-9c90-fc6ce500774f" (UID: "28ad1a21-b4f5-4af8-9c90-fc6ce500774f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:23:06.152628 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.152612 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-kube-api-access-8kqvn" pod "28ad1a21-b4f5-4af8-9c90-fc6ce500774f" (UID: "28ad1a21-b4f5-4af8-9c90-fc6ce500774f"). InnerVolumeSpecName "kube-api-access-8kqvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:23:06.152697 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.152609 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-metrics-certs" pod "28ad1a21-b4f5-4af8-9c90-fc6ce500774f" (UID: "28ad1a21-b4f5-4af8-9c90-fc6ce500774f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:23:06.251942 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.251907 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:23:06.251942 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.251936 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kqvn\" (UniqueName: \"kubernetes.io/projected/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-kube-api-access-8kqvn\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:23:06.251942 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.251946 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:23:06.252185 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.251956 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ad1a21-b4f5-4af8-9c90-fc6ce500774f-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:23:06.618663 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.618628 5463 generic.go:356] "Generic (PLEG): container finished" podID="28ad1a21-b4f5-4af8-9c90-fc6ce500774f" containerID="3adc87ede73663f345a147749eadefe4eb51d24a0ec47e61721538a49818581f" exitCode=0 Nov 03 03:23:06.618846 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.618689 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" Nov 03 03:23:06.618846 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.618714 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" event={"ID":"28ad1a21-b4f5-4af8-9c90-fc6ce500774f","Type":"ContainerDied","Data":"3adc87ede73663f345a147749eadefe4eb51d24a0ec47e61721538a49818581f"} Nov 03 03:23:06.618846 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.618772 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w" event={"ID":"28ad1a21-b4f5-4af8-9c90-fc6ce500774f","Type":"ContainerDied","Data":"c18ae6d3a5d11b506c8ade85df5193d91dafa78a0eab79aa25445c01b04806df"} Nov 03 03:23:06.618846 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.618794 5463 scope.go:122] "RemoveContainer" containerID="3adc87ede73663f345a147749eadefe4eb51d24a0ec47e61721538a49818581f" Nov 03 03:23:06.628263 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.628244 5463 scope.go:122] "RemoveContainer" containerID="3adc87ede73663f345a147749eadefe4eb51d24a0ec47e61721538a49818581f" Nov 03 03:23:06.628556 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:23:06.628529 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3adc87ede73663f345a147749eadefe4eb51d24a0ec47e61721538a49818581f\": container with ID starting with 3adc87ede73663f345a147749eadefe4eb51d24a0ec47e61721538a49818581f not found: ID does not exist" containerID="3adc87ede73663f345a147749eadefe4eb51d24a0ec47e61721538a49818581f" Nov 03 03:23:06.628622 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.628563 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3adc87ede73663f345a147749eadefe4eb51d24a0ec47e61721538a49818581f"} err="failed to get container status \"3adc87ede73663f345a147749eadefe4eb51d24a0ec47e61721538a49818581f\": rpc error: code = NotFound desc = could not find container \"3adc87ede73663f345a147749eadefe4eb51d24a0ec47e61721538a49818581f\": container with ID starting with 3adc87ede73663f345a147749eadefe4eb51d24a0ec47e61721538a49818581f not found: ID does not exist" Nov 03 03:23:06.783806 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.783725 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w"] Nov 03 03:23:06.785642 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:06.785622 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-bh42w"] Nov 03 03:23:07.422696 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:07.422661 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:23:07.656607 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:07.656563 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:23:08.046993 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:08.046964 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2"] Nov 03 03:23:08.097327 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:08.097299 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f6gw2"] Nov 03 03:23:08.280956 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:08.280926 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:23:08.442886 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:08.442853 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28ad1a21-b4f5-4af8-9c90-fc6ce500774f" path="/var/lib/kubelet/pods/28ad1a21-b4f5-4af8-9c90-fc6ce500774f/volumes" Nov 03 03:23:08.443305 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:08.443290 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c454515-5f30-4bdc-ad12-e0e2ff6bb51f" path="/var/lib/kubelet/pods/5c454515-5f30-4bdc-ad12-e0e2ff6bb51f/volumes" Nov 03 03:23:13.010799 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:23:13.010759 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:23:16.597114 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:16.597079 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:23:28.049013 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:28.048979 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq"] Nov 03 03:23:28.099270 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:28.099234 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d9lpq"] Nov 03 03:23:28.441407 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:28.441367 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3189d1cf-2221-4253-949d-4a976ca07ec3" path="/var/lib/kubelet/pods/3189d1cf-2221-4253-949d-4a976ca07ec3/volumes" Nov 03 03:23:32.993926 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:32.993894 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:23:39.048966 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:39.048881 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4"] Nov 03 03:23:39.098656 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:39.098622 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mz5n4"] Nov 03 03:23:40.442361 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:40.442324 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe62b8f-fc1b-4bcf-99cd-280030dff7d6" path="/var/lib/kubelet/pods/cbe62b8f-fc1b-4bcf-99cd-280030dff7d6/volumes" Nov 03 03:23:42.498441 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:23:42.498397 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:23:43.007641 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:23:43.007597 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:24:06.206435 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:06.206402 5463 scope.go:122] "RemoveContainer" containerID="22f87b55d92d7b333c9918210e730a3a0c80fc563ac42661e546cc470534f3bd" Nov 03 03:24:11.901539 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:11.901506 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6"] Nov 03 03:24:11.901937 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:11.901922 5463 cpu_manager.go:414] "containerMap: removing container" podUID="28ad1a21-b4f5-4af8-9c90-fc6ce500774f" containerName="maas-api" Nov 03 03:24:11.901937 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:11.901935 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ad1a21-b4f5-4af8-9c90-fc6ce500774f" containerName="maas-api" Nov 03 03:24:11.902054 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:11.902042 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ad1a21-b4f5-4af8-9c90-fc6ce500774f" containerName="maas-api" Nov 03 03:24:11.906717 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:11.906696 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6" Nov 03 03:24:11.956845 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:11.956817 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6"] Nov 03 03:24:12.005433 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:12.005404 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:24:12.005603 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:12.005492 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:24:12.009577 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:12.009557 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:24:12.009658 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:12.009582 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:24:12.014040 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:12.014023 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a332fc1c-f31b-4318-897d-0f2f8b14e889-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-cj4r6\" (UID: \"a332fc1c-f31b-4318-897d-0f2f8b14e889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6" Nov 03 03:24:12.014104 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:12.014070 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp8hr\" (UniqueName: \"kubernetes.io/projected/a332fc1c-f31b-4318-897d-0f2f8b14e889-kube-api-access-fp8hr\") pod \"maas-api-revoke-sub-452b6d7b-cj4r6\" (UID: \"a332fc1c-f31b-4318-897d-0f2f8b14e889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6" Nov 03 03:24:12.014198 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:12.014175 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a332fc1c-f31b-4318-897d-0f2f8b14e889-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-cj4r6\" (UID: \"a332fc1c-f31b-4318-897d-0f2f8b14e889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6" Nov 03 03:24:12.114814 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:12.114788 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a332fc1c-f31b-4318-897d-0f2f8b14e889-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-cj4r6\" (UID: \"a332fc1c-f31b-4318-897d-0f2f8b14e889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6" Nov 03 03:24:12.114991 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:12.114835 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp8hr\" (UniqueName: \"kubernetes.io/projected/a332fc1c-f31b-4318-897d-0f2f8b14e889-kube-api-access-fp8hr\") pod \"maas-api-revoke-sub-452b6d7b-cj4r6\" (UID: \"a332fc1c-f31b-4318-897d-0f2f8b14e889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6" Nov 03 03:24:12.114991 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:12.114885 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a332fc1c-f31b-4318-897d-0f2f8b14e889-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-cj4r6\" (UID: \"a332fc1c-f31b-4318-897d-0f2f8b14e889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6" Nov 03 03:24:12.115522 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:12.115497 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a332fc1c-f31b-4318-897d-0f2f8b14e889-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-cj4r6\" (UID: \"a332fc1c-f31b-4318-897d-0f2f8b14e889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6" Nov 03 03:24:12.117234 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:12.117213 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a332fc1c-f31b-4318-897d-0f2f8b14e889-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-cj4r6\" (UID: \"a332fc1c-f31b-4318-897d-0f2f8b14e889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6" Nov 03 03:24:12.175700 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:12.175624 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp8hr\" (UniqueName: \"kubernetes.io/projected/a332fc1c-f31b-4318-897d-0f2f8b14e889-kube-api-access-fp8hr\") pod \"maas-api-revoke-sub-452b6d7b-cj4r6\" (UID: \"a332fc1c-f31b-4318-897d-0f2f8b14e889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6" Nov 03 03:24:12.219565 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:12.219529 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6" Nov 03 03:24:12.591293 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:24:12.591258 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda332fc1c_f31b_4318_897d_0f2f8b14e889.slice/crio-6a4777dc090b8153337cf6abeeca19a09b1da9e6b325fd4fbe2a2bba66853f96 WatchSource:0}: Error finding container 6a4777dc090b8153337cf6abeeca19a09b1da9e6b325fd4fbe2a2bba66853f96: Status 404 returned error can't find the container with id 6a4777dc090b8153337cf6abeeca19a09b1da9e6b325fd4fbe2a2bba66853f96 Nov 03 03:24:12.592690 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:12.592659 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6"] Nov 03 03:24:13.006251 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:24:13.006211 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:24:13.053591 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:13.053558 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6" event={"ID":"a332fc1c-f31b-4318-897d-0f2f8b14e889","Type":"ContainerStarted","Data":"385469a204907a7e2b8e6ec1b869f959860cd99ff62f9b36438d89fa73f43fe4"} Nov 03 03:24:13.053733 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:13.053617 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6" event={"ID":"a332fc1c-f31b-4318-897d-0f2f8b14e889","Type":"ContainerStarted","Data":"6a4777dc090b8153337cf6abeeca19a09b1da9e6b325fd4fbe2a2bba66853f96"} Nov 03 03:24:13.168334 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:13.168288 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6" podStartSLOduration=2.168273764 podStartE2EDuration="2.168273764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:24:13.16721305 +0000 UTC m=+5534.851322202" watchObservedRunningTime="2026-11-03 03:24:13.168273764 +0000 UTC m=+5534.852382919" Nov 03 03:24:14.064097 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:14.064056 5463 generic.go:356] "Generic (PLEG): container finished" podID="a332fc1c-f31b-4318-897d-0f2f8b14e889" containerID="385469a204907a7e2b8e6ec1b869f959860cd99ff62f9b36438d89fa73f43fe4" exitCode=0 Nov 03 03:24:14.064626 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:14.064151 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6" event={"ID":"a332fc1c-f31b-4318-897d-0f2f8b14e889","Type":"ContainerDied","Data":"385469a204907a7e2b8e6ec1b869f959860cd99ff62f9b36438d89fa73f43fe4"} Nov 03 03:24:15.460644 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:15.460361 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6" Nov 03 03:24:15.556215 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:15.547223 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a332fc1c-f31b-4318-897d-0f2f8b14e889-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a332fc1c-f31b-4318-897d-0f2f8b14e889-maas-api-xks-ca\") pod \"a332fc1c-f31b-4318-897d-0f2f8b14e889\" (UID: \"a332fc1c-f31b-4318-897d-0f2f8b14e889\") " Nov 03 03:24:15.560306 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:15.560278 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a332fc1c-f31b-4318-897d-0f2f8b14e889-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a332fc1c-f31b-4318-897d-0f2f8b14e889-maas-api-openshift-ca\") pod \"a332fc1c-f31b-4318-897d-0f2f8b14e889\" (UID: \"a332fc1c-f31b-4318-897d-0f2f8b14e889\") " Nov 03 03:24:15.564956 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:15.564304 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a332fc1c-f31b-4318-897d-0f2f8b14e889-kube-api-access-fp8hr\" (UniqueName: \"kubernetes.io/projected/a332fc1c-f31b-4318-897d-0f2f8b14e889-kube-api-access-fp8hr\") pod \"a332fc1c-f31b-4318-897d-0f2f8b14e889\" (UID: \"a332fc1c-f31b-4318-897d-0f2f8b14e889\") " Nov 03 03:24:15.594414 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:15.579916 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a332fc1c-f31b-4318-897d-0f2f8b14e889-kube-api-access-fp8hr" pod "a332fc1c-f31b-4318-897d-0f2f8b14e889" (UID: "a332fc1c-f31b-4318-897d-0f2f8b14e889"). InnerVolumeSpecName "kube-api-access-fp8hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:24:15.594414 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:15.588384 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a332fc1c-f31b-4318-897d-0f2f8b14e889-maas-api-openshift-ca" pod "a332fc1c-f31b-4318-897d-0f2f8b14e889" (UID: "a332fc1c-f31b-4318-897d-0f2f8b14e889"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:24:15.598739 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:15.598713 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a332fc1c-f31b-4318-897d-0f2f8b14e889-maas-api-xks-ca" pod "a332fc1c-f31b-4318-897d-0f2f8b14e889" (UID: "a332fc1c-f31b-4318-897d-0f2f8b14e889"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:24:15.666213 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:15.665927 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a332fc1c-f31b-4318-897d-0f2f8b14e889-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:24:15.666213 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:15.665972 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a332fc1c-f31b-4318-897d-0f2f8b14e889-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:24:15.666213 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:15.665989 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp8hr\" (UniqueName: \"kubernetes.io/projected/a332fc1c-f31b-4318-897d-0f2f8b14e889-kube-api-access-fp8hr\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:24:16.117214 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:16.117107 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6" event={"ID":"a332fc1c-f31b-4318-897d-0f2f8b14e889","Type":"ContainerDied","Data":"6a4777dc090b8153337cf6abeeca19a09b1da9e6b325fd4fbe2a2bba66853f96"} Nov 03 03:24:16.117214 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:16.117207 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a4777dc090b8153337cf6abeeca19a09b1da9e6b325fd4fbe2a2bba66853f96" Nov 03 03:24:16.117508 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:16.117295 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6" Nov 03 03:24:24.763122 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:24.763083 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:24:31.319446 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:31.319338 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:24:33.313426 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:33.313389 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:24:35.454444 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:35.454406 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh"] Nov 03 03:24:35.454825 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:35.454808 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a332fc1c-f31b-4318-897d-0f2f8b14e889" containerName="revoke-keys" Nov 03 03:24:35.454825 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:35.454819 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a332fc1c-f31b-4318-897d-0f2f8b14e889" containerName="revoke-keys" Nov 03 03:24:35.454914 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:35.454903 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a332fc1c-f31b-4318-897d-0f2f8b14e889" containerName="revoke-keys" Nov 03 03:24:35.458369 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:35.458352 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh" Nov 03 03:24:35.511790 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:35.511758 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh"] Nov 03 03:24:35.534893 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:35.534863 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77654092-ca40-429e-a5b9-68ea634c5d48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-sz4wh\" (UID: \"77654092-ca40-429e-a5b9-68ea634c5d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh" Nov 03 03:24:35.535057 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:35.535030 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbq5p\" (UniqueName: \"kubernetes.io/projected/77654092-ca40-429e-a5b9-68ea634c5d48-kube-api-access-rbq5p\") pod \"maas-api-revoke-sub-3e31a506-sz4wh\" (UID: \"77654092-ca40-429e-a5b9-68ea634c5d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh" Nov 03 03:24:35.535247 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:35.535227 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77654092-ca40-429e-a5b9-68ea634c5d48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-sz4wh\" (UID: \"77654092-ca40-429e-a5b9-68ea634c5d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh" Nov 03 03:24:35.558937 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:35.558912 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:24:35.559037 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:35.558975 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:24:35.569069 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:35.569052 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:24:35.569069 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:35.569055 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:24:35.636589 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:35.636557 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77654092-ca40-429e-a5b9-68ea634c5d48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-sz4wh\" (UID: \"77654092-ca40-429e-a5b9-68ea634c5d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh" Nov 03 03:24:35.636724 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:35.636638 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77654092-ca40-429e-a5b9-68ea634c5d48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-sz4wh\" (UID: \"77654092-ca40-429e-a5b9-68ea634c5d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh" Nov 03 03:24:35.636724 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:35.636671 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbq5p\" (UniqueName: \"kubernetes.io/projected/77654092-ca40-429e-a5b9-68ea634c5d48-kube-api-access-rbq5p\") pod \"maas-api-revoke-sub-3e31a506-sz4wh\" (UID: \"77654092-ca40-429e-a5b9-68ea634c5d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh" Nov 03 03:24:35.637351 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:35.637330 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77654092-ca40-429e-a5b9-68ea634c5d48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-sz4wh\" (UID: \"77654092-ca40-429e-a5b9-68ea634c5d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh" Nov 03 03:24:35.638871 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:35.638849 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77654092-ca40-429e-a5b9-68ea634c5d48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-sz4wh\" (UID: \"77654092-ca40-429e-a5b9-68ea634c5d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh" Nov 03 03:24:35.694085 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:35.694057 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbq5p\" (UniqueName: \"kubernetes.io/projected/77654092-ca40-429e-a5b9-68ea634c5d48-kube-api-access-rbq5p\") pod \"maas-api-revoke-sub-3e31a506-sz4wh\" (UID: \"77654092-ca40-429e-a5b9-68ea634c5d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh" Nov 03 03:24:35.771184 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:35.771125 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh" Nov 03 03:24:36.143621 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:24:36.143587 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77654092_ca40_429e_a5b9_68ea634c5d48.slice/crio-776026e50ca34e502143345090443162a0216fa7d8dd959c2f4fe4f565130249 WatchSource:0}: Error finding container 776026e50ca34e502143345090443162a0216fa7d8dd959c2f4fe4f565130249: Status 404 returned error can't find the container with id 776026e50ca34e502143345090443162a0216fa7d8dd959c2f4fe4f565130249 Nov 03 03:24:36.145536 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:36.145514 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh"] Nov 03 03:24:36.260957 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:36.260922 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh" event={"ID":"77654092-ca40-429e-a5b9-68ea634c5d48","Type":"ContainerStarted","Data":"776026e50ca34e502143345090443162a0216fa7d8dd959c2f4fe4f565130249"} Nov 03 03:24:37.271313 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:37.271180 5463 generic.go:356] "Generic (PLEG): container finished" podID="77654092-ca40-429e-a5b9-68ea634c5d48" containerID="25fae792f6076283644db9aedee1eb9966f64e054cd7a2c19e22fdd5cb5f2205" exitCode=0 Nov 03 03:24:37.271313 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:37.271242 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh" event={"ID":"77654092-ca40-429e-a5b9-68ea634c5d48","Type":"ContainerDied","Data":"25fae792f6076283644db9aedee1eb9966f64e054cd7a2c19e22fdd5cb5f2205"} Nov 03 03:24:38.728360 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:38.728328 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh" Nov 03 03:24:38.732091 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:38.732059 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77654092-ca40-429e-a5b9-68ea634c5d48-kube-api-access-rbq5p\" (UniqueName: \"kubernetes.io/projected/77654092-ca40-429e-a5b9-68ea634c5d48-kube-api-access-rbq5p\") pod \"77654092-ca40-429e-a5b9-68ea634c5d48\" (UID: \"77654092-ca40-429e-a5b9-68ea634c5d48\") " Nov 03 03:24:38.732269 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:38.732236 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77654092-ca40-429e-a5b9-68ea634c5d48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77654092-ca40-429e-a5b9-68ea634c5d48-maas-api-openshift-ca\") pod \"77654092-ca40-429e-a5b9-68ea634c5d48\" (UID: \"77654092-ca40-429e-a5b9-68ea634c5d48\") " Nov 03 03:24:38.732356 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:38.732320 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77654092-ca40-429e-a5b9-68ea634c5d48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77654092-ca40-429e-a5b9-68ea634c5d48-maas-api-xks-ca\") pod \"77654092-ca40-429e-a5b9-68ea634c5d48\" (UID: \"77654092-ca40-429e-a5b9-68ea634c5d48\") " Nov 03 03:24:38.732617 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:38.732588 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77654092-ca40-429e-a5b9-68ea634c5d48-maas-api-openshift-ca" pod "77654092-ca40-429e-a5b9-68ea634c5d48" (UID: "77654092-ca40-429e-a5b9-68ea634c5d48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:24:38.732777 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:38.732752 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77654092-ca40-429e-a5b9-68ea634c5d48-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:24:38.734410 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:38.734377 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77654092-ca40-429e-a5b9-68ea634c5d48-kube-api-access-rbq5p" pod "77654092-ca40-429e-a5b9-68ea634c5d48" (UID: "77654092-ca40-429e-a5b9-68ea634c5d48"). InnerVolumeSpecName "kube-api-access-rbq5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:24:38.735683 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:38.735663 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77654092-ca40-429e-a5b9-68ea634c5d48-maas-api-xks-ca" pod "77654092-ca40-429e-a5b9-68ea634c5d48" (UID: "77654092-ca40-429e-a5b9-68ea634c5d48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:24:38.833903 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:38.833858 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbq5p\" (UniqueName: \"kubernetes.io/projected/77654092-ca40-429e-a5b9-68ea634c5d48-kube-api-access-rbq5p\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:24:38.833903 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:38.833903 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77654092-ca40-429e-a5b9-68ea634c5d48-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:24:39.318130 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:39.318098 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh" Nov 03 03:24:39.318468 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:39.318163 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh" event={"ID":"77654092-ca40-429e-a5b9-68ea634c5d48","Type":"ContainerDied","Data":"776026e50ca34e502143345090443162a0216fa7d8dd959c2f4fe4f565130249"} Nov 03 03:24:39.318573 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:39.318504 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="776026e50ca34e502143345090443162a0216fa7d8dd959c2f4fe4f565130249" Nov 03 03:24:43.008045 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:24:43.008010 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:24:59.418383 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:59.418344 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm"] Nov 03 03:24:59.418797 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:59.418762 5463 cpu_manager.go:414] "containerMap: removing container" podUID="77654092-ca40-429e-a5b9-68ea634c5d48" containerName="revoke-keys" Nov 03 03:24:59.418797 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:59.418774 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77654092-ca40-429e-a5b9-68ea634c5d48" containerName="revoke-keys" Nov 03 03:24:59.418890 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:59.418866 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="77654092-ca40-429e-a5b9-68ea634c5d48" containerName="revoke-keys" Nov 03 03:24:59.421764 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:59.421748 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm" Nov 03 03:24:59.458291 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:59.458263 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e070a2f-881a-40da-95f4-bc88bd52b096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1eb55f9d-qspfm\" (UID: \"2e070a2f-881a-40da-95f4-bc88bd52b096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm" Nov 03 03:24:59.458420 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:59.458332 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dh2j\" (UniqueName: \"kubernetes.io/projected/2e070a2f-881a-40da-95f4-bc88bd52b096-kube-api-access-7dh2j\") pod \"maas-api-revoke-sub-1eb55f9d-qspfm\" (UID: \"2e070a2f-881a-40da-95f4-bc88bd52b096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm" Nov 03 03:24:59.458465 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:59.458427 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e070a2f-881a-40da-95f4-bc88bd52b096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1eb55f9d-qspfm\" (UID: \"2e070a2f-881a-40da-95f4-bc88bd52b096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm" Nov 03 03:24:59.478610 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:59.478586 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm"] Nov 03 03:24:59.526561 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:59.526532 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:24:59.526561 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:59.526527 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:24:59.526751 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:59.526598 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:24:59.526751 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:59.526607 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:24:59.559167 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:59.559138 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e070a2f-881a-40da-95f4-bc88bd52b096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1eb55f9d-qspfm\" (UID: \"2e070a2f-881a-40da-95f4-bc88bd52b096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm" Nov 03 03:24:59.559311 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:59.559223 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dh2j\" (UniqueName: \"kubernetes.io/projected/2e070a2f-881a-40da-95f4-bc88bd52b096-kube-api-access-7dh2j\") pod \"maas-api-revoke-sub-1eb55f9d-qspfm\" (UID: \"2e070a2f-881a-40da-95f4-bc88bd52b096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm" Nov 03 03:24:59.559367 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:59.559317 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e070a2f-881a-40da-95f4-bc88bd52b096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1eb55f9d-qspfm\" (UID: \"2e070a2f-881a-40da-95f4-bc88bd52b096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm" Nov 03 03:24:59.559850 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:59.559829 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e070a2f-881a-40da-95f4-bc88bd52b096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1eb55f9d-qspfm\" (UID: \"2e070a2f-881a-40da-95f4-bc88bd52b096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm" Nov 03 03:24:59.561629 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:59.561608 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e070a2f-881a-40da-95f4-bc88bd52b096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1eb55f9d-qspfm\" (UID: \"2e070a2f-881a-40da-95f4-bc88bd52b096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm" Nov 03 03:24:59.616317 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:59.616293 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dh2j\" (UniqueName: \"kubernetes.io/projected/2e070a2f-881a-40da-95f4-bc88bd52b096-kube-api-access-7dh2j\") pod \"maas-api-revoke-sub-1eb55f9d-qspfm\" (UID: \"2e070a2f-881a-40da-95f4-bc88bd52b096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm" Nov 03 03:24:59.734465 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:24:59.734407 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm" Nov 03 03:25:00.102427 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:25:00.102395 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e070a2f_881a_40da_95f4_bc88bd52b096.slice/crio-06a75554b6077c36bfd357fa5e50a1646691bfb5f5050a9b0aa8763fcbce8378 WatchSource:0}: Error finding container 06a75554b6077c36bfd357fa5e50a1646691bfb5f5050a9b0aa8763fcbce8378: Status 404 returned error can't find the container with id 06a75554b6077c36bfd357fa5e50a1646691bfb5f5050a9b0aa8763fcbce8378 Nov 03 03:25:00.104625 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:00.104596 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm"] Nov 03 03:25:00.462388 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:00.462353 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm" event={"ID":"2e070a2f-881a-40da-95f4-bc88bd52b096","Type":"ContainerStarted","Data":"dc84199a6f4d7d2857db1b94379fcdd6730889ee689b9c52713c701494fc0f4e"} Nov 03 03:25:00.462745 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:00.462409 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm" event={"ID":"2e070a2f-881a-40da-95f4-bc88bd52b096","Type":"ContainerStarted","Data":"06a75554b6077c36bfd357fa5e50a1646691bfb5f5050a9b0aa8763fcbce8378"} Nov 03 03:25:00.576710 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:00.576663 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm" podStartSLOduration=1.576649744 podStartE2EDuration="1.576649744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:00.576013519 +0000 UTC m=+5582.260122675" watchObservedRunningTime="2026-11-03 03:25:00.576649744 +0000 UTC m=+5582.260758899" Nov 03 03:25:01.469930 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:01.469897 5463 generic.go:356] "Generic (PLEG): container finished" podID="2e070a2f-881a-40da-95f4-bc88bd52b096" containerID="dc84199a6f4d7d2857db1b94379fcdd6730889ee689b9c52713c701494fc0f4e" exitCode=0 Nov 03 03:25:01.470346 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:01.469947 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm" event={"ID":"2e070a2f-881a-40da-95f4-bc88bd52b096","Type":"ContainerDied","Data":"dc84199a6f4d7d2857db1b94379fcdd6730889ee689b9c52713c701494fc0f4e"} Nov 03 03:25:01.614362 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:01.614327 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:25:01.765879 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:01.765800 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:25:02.608512 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:02.608489 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm" Nov 03 03:25:02.688464 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:02.688440 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e070a2f-881a-40da-95f4-bc88bd52b096-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e070a2f-881a-40da-95f4-bc88bd52b096-maas-api-openshift-ca\") pod \"2e070a2f-881a-40da-95f4-bc88bd52b096\" (UID: \"2e070a2f-881a-40da-95f4-bc88bd52b096\") " Nov 03 03:25:02.688619 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:02.688491 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e070a2f-881a-40da-95f4-bc88bd52b096-kube-api-access-7dh2j\" (UniqueName: \"kubernetes.io/projected/2e070a2f-881a-40da-95f4-bc88bd52b096-kube-api-access-7dh2j\") pod \"2e070a2f-881a-40da-95f4-bc88bd52b096\" (UID: \"2e070a2f-881a-40da-95f4-bc88bd52b096\") " Nov 03 03:25:02.688619 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:02.688529 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e070a2f-881a-40da-95f4-bc88bd52b096-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e070a2f-881a-40da-95f4-bc88bd52b096-maas-api-xks-ca\") pod \"2e070a2f-881a-40da-95f4-bc88bd52b096\" (UID: \"2e070a2f-881a-40da-95f4-bc88bd52b096\") " Nov 03 03:25:02.688794 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:02.688769 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e070a2f-881a-40da-95f4-bc88bd52b096-maas-api-openshift-ca" pod "2e070a2f-881a-40da-95f4-bc88bd52b096" (UID: "2e070a2f-881a-40da-95f4-bc88bd52b096"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:25:02.688881 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:02.688865 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e070a2f-881a-40da-95f4-bc88bd52b096-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:25:02.690586 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:02.690565 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e070a2f-881a-40da-95f4-bc88bd52b096-maas-api-xks-ca" pod "2e070a2f-881a-40da-95f4-bc88bd52b096" (UID: "2e070a2f-881a-40da-95f4-bc88bd52b096"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:25:02.690709 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:02.690565 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e070a2f-881a-40da-95f4-bc88bd52b096-kube-api-access-7dh2j" pod "2e070a2f-881a-40da-95f4-bc88bd52b096" (UID: "2e070a2f-881a-40da-95f4-bc88bd52b096"). InnerVolumeSpecName "kube-api-access-7dh2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:25:02.789800 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:02.789740 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dh2j\" (UniqueName: \"kubernetes.io/projected/2e070a2f-881a-40da-95f4-bc88bd52b096-kube-api-access-7dh2j\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:25:02.789800 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:02.789765 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e070a2f-881a-40da-95f4-bc88bd52b096-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:25:03.481501 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:03.481473 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm" Nov 03 03:25:03.481501 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:03.481495 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm" event={"ID":"2e070a2f-881a-40da-95f4-bc88bd52b096","Type":"ContainerDied","Data":"06a75554b6077c36bfd357fa5e50a1646691bfb5f5050a9b0aa8763fcbce8378"} Nov 03 03:25:03.481767 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:03.481543 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06a75554b6077c36bfd357fa5e50a1646691bfb5f5050a9b0aa8763fcbce8378" Nov 03 03:25:06.234954 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:06.234919 5463 scope.go:122] "RemoveContainer" containerID="e172dd191d03c64376ece83d4f53f65785ea02d113f9145f4a2203ee2dedfd92" Nov 03 03:25:06.243992 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:06.243969 5463 scope.go:122] "RemoveContainer" containerID="7347f46639e287331054ef663b12b31fccc8f29b665a1cff59f8fb7718c31ba2" Nov 03 03:25:13.005748 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:25:13.005674 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:25:16.313564 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:16.313521 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-595669665d-rpgr6"] Nov 03 03:25:16.314077 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:16.313965 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2e070a2f-881a-40da-95f4-bc88bd52b096" containerName="revoke-keys" Nov 03 03:25:16.314077 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:16.313980 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e070a2f-881a-40da-95f4-bc88bd52b096" containerName="revoke-keys" Nov 03 03:25:16.314077 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:16.314072 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e070a2f-881a-40da-95f4-bc88bd52b096" containerName="revoke-keys" Nov 03 03:25:16.317143 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:16.317126 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-595669665d-rpgr6" Nov 03 03:25:16.370610 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:16.370582 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-595669665d-rpgr6"] Nov 03 03:25:16.416980 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:16.416951 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef792ca8-5cab-4b3a-8417-6708e36bff96-plugins-config-volume\") pod \"payload-pre-processing-595669665d-rpgr6\" (UID: \"ef792ca8-5cab-4b3a-8417-6708e36bff96\") " pod="openshift-ingress/payload-pre-processing-595669665d-rpgr6" Nov 03 03:25:16.417143 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:16.417082 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fjf4\" (UniqueName: \"kubernetes.io/projected/ef792ca8-5cab-4b3a-8417-6708e36bff96-kube-api-access-4fjf4\") pod \"payload-pre-processing-595669665d-rpgr6\" (UID: \"ef792ca8-5cab-4b3a-8417-6708e36bff96\") " pod="openshift-ingress/payload-pre-processing-595669665d-rpgr6" Nov 03 03:25:16.518486 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:16.518450 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fjf4\" (UniqueName: \"kubernetes.io/projected/ef792ca8-5cab-4b3a-8417-6708e36bff96-kube-api-access-4fjf4\") pod \"payload-pre-processing-595669665d-rpgr6\" (UID: \"ef792ca8-5cab-4b3a-8417-6708e36bff96\") " pod="openshift-ingress/payload-pre-processing-595669665d-rpgr6" Nov 03 03:25:16.518686 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:16.518519 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef792ca8-5cab-4b3a-8417-6708e36bff96-plugins-config-volume\") pod \"payload-pre-processing-595669665d-rpgr6\" (UID: \"ef792ca8-5cab-4b3a-8417-6708e36bff96\") " pod="openshift-ingress/payload-pre-processing-595669665d-rpgr6" Nov 03 03:25:16.519260 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:16.519241 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef792ca8-5cab-4b3a-8417-6708e36bff96-plugins-config-volume\") pod \"payload-pre-processing-595669665d-rpgr6\" (UID: \"ef792ca8-5cab-4b3a-8417-6708e36bff96\") " pod="openshift-ingress/payload-pre-processing-595669665d-rpgr6" Nov 03 03:25:16.575861 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:16.575792 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fjf4\" (UniqueName: \"kubernetes.io/projected/ef792ca8-5cab-4b3a-8417-6708e36bff96-kube-api-access-4fjf4\") pod \"payload-pre-processing-595669665d-rpgr6\" (UID: \"ef792ca8-5cab-4b3a-8417-6708e36bff96\") " pod="openshift-ingress/payload-pre-processing-595669665d-rpgr6" Nov 03 03:25:16.629769 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:16.629744 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-595669665d-rpgr6" Nov 03 03:25:16.995231 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:25:16.995185 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef792ca8_5cab_4b3a_8417_6708e36bff96.slice/crio-da5020cea7e647f6b8ff00e6994172b11227a40674911c95981e4730bb3039c3 WatchSource:0}: Error finding container da5020cea7e647f6b8ff00e6994172b11227a40674911c95981e4730bb3039c3: Status 404 returned error can't find the container with id da5020cea7e647f6b8ff00e6994172b11227a40674911c95981e4730bb3039c3 Nov 03 03:25:16.996884 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:16.996848 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-595669665d-rpgr6"] Nov 03 03:25:17.557378 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:17.557337 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-595669665d-rpgr6" event={"ID":"ef792ca8-5cab-4b3a-8417-6708e36bff96","Type":"ContainerStarted","Data":"70efc4d16dbb53b3c76de1da2bd8cb8f991240748940fcd5b996af44cad2f8c0"} Nov 03 03:25:17.557851 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:17.557414 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-595669665d-rpgr6" event={"ID":"ef792ca8-5cab-4b3a-8417-6708e36bff96","Type":"ContainerStarted","Data":"da5020cea7e647f6b8ff00e6994172b11227a40674911c95981e4730bb3039c3"} Nov 03 03:25:17.557851 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:17.557480 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-595669665d-rpgr6" Nov 03 03:25:17.674484 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:17.674432 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-595669665d-rpgr6" podStartSLOduration=1.674418965 podStartE2EDuration="1.674418965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:17.671713238 +0000 UTC m=+5599.355822393" watchObservedRunningTime="2026-11-03 03:25:17.674418965 +0000 UTC m=+5599.358528127" Nov 03 03:25:28.567720 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:28.567692 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-595669665d-rpgr6" Nov 03 03:25:28.726164 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:28.726136 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h"] Nov 03 03:25:30.827667 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:30.827626 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55c996f8f7-zx4vc"] Nov 03 03:25:30.831453 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:30.831435 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55c996f8f7-zx4vc" Nov 03 03:25:30.884723 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:30.884685 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55c996f8f7-zx4vc"] Nov 03 03:25:30.960921 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:30.960883 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mck45\" (UniqueName: \"kubernetes.io/projected/af6b24f1-1697-4515-9472-16dac5d82805-kube-api-access-mck45\") pod \"payload-processing-55c996f8f7-zx4vc\" (UID: \"af6b24f1-1697-4515-9472-16dac5d82805\") " pod="openshift-ingress/payload-processing-55c996f8f7-zx4vc" Nov 03 03:25:30.961130 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:30.960934 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af6b24f1-1697-4515-9472-16dac5d82805-plugins-config-volume\") pod \"payload-processing-55c996f8f7-zx4vc\" (UID: \"af6b24f1-1697-4515-9472-16dac5d82805\") " pod="openshift-ingress/payload-processing-55c996f8f7-zx4vc" Nov 03 03:25:31.062540 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:31.062511 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mck45\" (UniqueName: \"kubernetes.io/projected/af6b24f1-1697-4515-9472-16dac5d82805-kube-api-access-mck45\") pod \"payload-processing-55c996f8f7-zx4vc\" (UID: \"af6b24f1-1697-4515-9472-16dac5d82805\") " pod="openshift-ingress/payload-processing-55c996f8f7-zx4vc" Nov 03 03:25:31.062669 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:31.062562 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af6b24f1-1697-4515-9472-16dac5d82805-plugins-config-volume\") pod \"payload-processing-55c996f8f7-zx4vc\" (UID: \"af6b24f1-1697-4515-9472-16dac5d82805\") " pod="openshift-ingress/payload-processing-55c996f8f7-zx4vc" Nov 03 03:25:31.063321 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:31.063302 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af6b24f1-1697-4515-9472-16dac5d82805-plugins-config-volume\") pod \"payload-processing-55c996f8f7-zx4vc\" (UID: \"af6b24f1-1697-4515-9472-16dac5d82805\") " pod="openshift-ingress/payload-processing-55c996f8f7-zx4vc" Nov 03 03:25:31.119977 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:31.119908 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mck45\" (UniqueName: \"kubernetes.io/projected/af6b24f1-1697-4515-9472-16dac5d82805-kube-api-access-mck45\") pod \"payload-processing-55c996f8f7-zx4vc\" (UID: \"af6b24f1-1697-4515-9472-16dac5d82805\") " pod="openshift-ingress/payload-processing-55c996f8f7-zx4vc" Nov 03 03:25:31.143602 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:31.143580 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55c996f8f7-zx4vc" Nov 03 03:25:31.511823 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:25:31.511782 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf6b24f1_1697_4515_9472_16dac5d82805.slice/crio-13a994a718bfde39110c1cdbb6e38897c2d22d3315664e87f591e1fb531edd33 WatchSource:0}: Error finding container 13a994a718bfde39110c1cdbb6e38897c2d22d3315664e87f591e1fb531edd33: Status 404 returned error can't find the container with id 13a994a718bfde39110c1cdbb6e38897c2d22d3315664e87f591e1fb531edd33 Nov 03 03:25:31.514082 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:31.514059 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55c996f8f7-zx4vc"] Nov 03 03:25:31.634693 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:31.634658 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55c996f8f7-zx4vc" event={"ID":"af6b24f1-1697-4515-9472-16dac5d82805","Type":"ContainerStarted","Data":"9fba5b81f0914614dbcc970ff5f803659c49d159ca84889a87b65a6fe76e0135"} Nov 03 03:25:31.634805 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:31.634725 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55c996f8f7-zx4vc" Nov 03 03:25:31.634805 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:31.634747 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55c996f8f7-zx4vc" event={"ID":"af6b24f1-1697-4515-9472-16dac5d82805","Type":"ContainerStarted","Data":"13a994a718bfde39110c1cdbb6e38897c2d22d3315664e87f591e1fb531edd33"} Nov 03 03:25:31.748940 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:31.748896 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55c996f8f7-zx4vc" podStartSLOduration=1.748882206 podStartE2EDuration="1.748882206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:31.747491164 +0000 UTC m=+5613.431600321" watchObservedRunningTime="2026-11-03 03:25:31.748882206 +0000 UTC m=+5613.432991418" Nov 03 03:25:33.650242 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:33.650187 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:25:33.727325 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:33.727287 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h" podUID="7b5071c8-5a0d-46cd-bd1e-28d5632cf091" containerName="payload-pre-processing" containerID="cri-o://e7b24e7b96683d15e18170239db563aba36d0784c7ac08457f955355fee48c86" gracePeriod=25 Nov 03 03:25:33.984775 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:33.984747 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h" Nov 03 03:25:34.090267 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:34.090244 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b5071c8-5a0d-46cd-bd1e-28d5632cf091-kube-api-access-g5m2g\" (UniqueName: \"kubernetes.io/projected/7b5071c8-5a0d-46cd-bd1e-28d5632cf091-kube-api-access-g5m2g\") pod \"7b5071c8-5a0d-46cd-bd1e-28d5632cf091\" (UID: \"7b5071c8-5a0d-46cd-bd1e-28d5632cf091\") " Nov 03 03:25:34.090438 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:34.090371 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b5071c8-5a0d-46cd-bd1e-28d5632cf091-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b5071c8-5a0d-46cd-bd1e-28d5632cf091-plugins-config-volume\") pod \"7b5071c8-5a0d-46cd-bd1e-28d5632cf091\" (UID: \"7b5071c8-5a0d-46cd-bd1e-28d5632cf091\") " Nov 03 03:25:34.090721 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:34.090700 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b5071c8-5a0d-46cd-bd1e-28d5632cf091-plugins-config-volume" pod "7b5071c8-5a0d-46cd-bd1e-28d5632cf091" (UID: "7b5071c8-5a0d-46cd-bd1e-28d5632cf091"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:25:34.092258 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:34.092239 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5071c8-5a0d-46cd-bd1e-28d5632cf091-kube-api-access-g5m2g" pod "7b5071c8-5a0d-46cd-bd1e-28d5632cf091" (UID: "7b5071c8-5a0d-46cd-bd1e-28d5632cf091"). InnerVolumeSpecName "kube-api-access-g5m2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:25:34.191420 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:34.191382 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b5071c8-5a0d-46cd-bd1e-28d5632cf091-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:25:34.191420 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:34.191411 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5m2g\" (UniqueName: \"kubernetes.io/projected/7b5071c8-5a0d-46cd-bd1e-28d5632cf091-kube-api-access-g5m2g\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:25:34.650270 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:34.650238 5463 generic.go:356] "Generic (PLEG): container finished" podID="7b5071c8-5a0d-46cd-bd1e-28d5632cf091" containerID="e7b24e7b96683d15e18170239db563aba36d0784c7ac08457f955355fee48c86" exitCode=0 Nov 03 03:25:34.650671 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:34.650276 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h" event={"ID":"7b5071c8-5a0d-46cd-bd1e-28d5632cf091","Type":"ContainerDied","Data":"e7b24e7b96683d15e18170239db563aba36d0784c7ac08457f955355fee48c86"} Nov 03 03:25:34.650671 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:34.650289 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h" Nov 03 03:25:34.650671 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:34.650316 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h" event={"ID":"7b5071c8-5a0d-46cd-bd1e-28d5632cf091","Type":"ContainerDied","Data":"dd6c19d27e543ccebdaa93659ca3bda992f7985f44b131ff2fc6d9553be6760a"} Nov 03 03:25:34.650671 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:34.650337 5463 scope.go:122] "RemoveContainer" containerID="e7b24e7b96683d15e18170239db563aba36d0784c7ac08457f955355fee48c86" Nov 03 03:25:34.659254 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:34.659238 5463 scope.go:122] "RemoveContainer" containerID="e7b24e7b96683d15e18170239db563aba36d0784c7ac08457f955355fee48c86" Nov 03 03:25:34.659511 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:25:34.659493 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b24e7b96683d15e18170239db563aba36d0784c7ac08457f955355fee48c86\": container with ID starting with e7b24e7b96683d15e18170239db563aba36d0784c7ac08457f955355fee48c86 not found: ID does not exist" containerID="e7b24e7b96683d15e18170239db563aba36d0784c7ac08457f955355fee48c86" Nov 03 03:25:34.659562 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:34.659518 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b24e7b96683d15e18170239db563aba36d0784c7ac08457f955355fee48c86"} err="failed to get container status \"e7b24e7b96683d15e18170239db563aba36d0784c7ac08457f955355fee48c86\": rpc error: code = NotFound desc = could not find container \"e7b24e7b96683d15e18170239db563aba36d0784c7ac08457f955355fee48c86\": container with ID starting with e7b24e7b96683d15e18170239db563aba36d0784c7ac08457f955355fee48c86 not found: ID does not exist" Nov 03 03:25:34.815064 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:34.814991 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h"] Nov 03 03:25:34.865569 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:34.865541 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-lsc6h"] Nov 03 03:25:36.441670 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:36.441638 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b5071c8-5a0d-46cd-bd1e-28d5632cf091" path="/var/lib/kubelet/pods/7b5071c8-5a0d-46cd-bd1e-28d5632cf091/volumes" Nov 03 03:25:42.643074 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:42.643045 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55c996f8f7-zx4vc" Nov 03 03:25:43.005554 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:25:43.005518 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:25:45.177508 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:45.177470 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b96c95569-z2wqs"] Nov 03 03:25:45.177979 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:45.177878 5463 cpu_manager.go:414] "containerMap: removing container" podUID="7b5071c8-5a0d-46cd-bd1e-28d5632cf091" containerName="payload-pre-processing" Nov 03 03:25:45.177979 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:45.177890 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b5071c8-5a0d-46cd-bd1e-28d5632cf091" containerName="payload-pre-processing" Nov 03 03:25:45.178059 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:45.178003 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b5071c8-5a0d-46cd-bd1e-28d5632cf091" containerName="payload-pre-processing" Nov 03 03:25:45.182332 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:45.182311 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b96c95569-z2wqs" Nov 03 03:25:45.233554 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:45.233530 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b96c95569-z2wqs"] Nov 03 03:25:45.302811 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:45.302779 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9w4p\" (UniqueName: \"kubernetes.io/projected/55406f41-e426-43a4-b621-7a38b12fe150-kube-api-access-s9w4p\") pod \"payload-processing-5b96c95569-z2wqs\" (UID: \"55406f41-e426-43a4-b621-7a38b12fe150\") " pod="openshift-ingress/payload-processing-5b96c95569-z2wqs" Nov 03 03:25:45.302811 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:45.302819 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55406f41-e426-43a4-b621-7a38b12fe150-plugins-config-volume\") pod \"payload-processing-5b96c95569-z2wqs\" (UID: \"55406f41-e426-43a4-b621-7a38b12fe150\") " pod="openshift-ingress/payload-processing-5b96c95569-z2wqs" Nov 03 03:25:45.404003 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:45.403971 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9w4p\" (UniqueName: \"kubernetes.io/projected/55406f41-e426-43a4-b621-7a38b12fe150-kube-api-access-s9w4p\") pod \"payload-processing-5b96c95569-z2wqs\" (UID: \"55406f41-e426-43a4-b621-7a38b12fe150\") " pod="openshift-ingress/payload-processing-5b96c95569-z2wqs" Nov 03 03:25:45.404166 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:45.404008 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55406f41-e426-43a4-b621-7a38b12fe150-plugins-config-volume\") pod \"payload-processing-5b96c95569-z2wqs\" (UID: \"55406f41-e426-43a4-b621-7a38b12fe150\") " pod="openshift-ingress/payload-processing-5b96c95569-z2wqs" Nov 03 03:25:45.404749 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:45.404727 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55406f41-e426-43a4-b621-7a38b12fe150-plugins-config-volume\") pod \"payload-processing-5b96c95569-z2wqs\" (UID: \"55406f41-e426-43a4-b621-7a38b12fe150\") " pod="openshift-ingress/payload-processing-5b96c95569-z2wqs" Nov 03 03:25:45.462261 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:45.462173 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9w4p\" (UniqueName: \"kubernetes.io/projected/55406f41-e426-43a4-b621-7a38b12fe150-kube-api-access-s9w4p\") pod \"payload-processing-5b96c95569-z2wqs\" (UID: \"55406f41-e426-43a4-b621-7a38b12fe150\") " pod="openshift-ingress/payload-processing-5b96c95569-z2wqs" Nov 03 03:25:45.495766 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:45.495736 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b96c95569-z2wqs" Nov 03 03:25:45.861957 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:25:45.861915 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55406f41_e426_43a4_b621_7a38b12fe150.slice/crio-8f006474e7b750ef4ab887633333299bb895d60e3e81cce89a35a6392909be76 WatchSource:0}: Error finding container 8f006474e7b750ef4ab887633333299bb895d60e3e81cce89a35a6392909be76: Status 404 returned error can't find the container with id 8f006474e7b750ef4ab887633333299bb895d60e3e81cce89a35a6392909be76 Nov 03 03:25:45.863040 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:45.863009 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b96c95569-z2wqs"] Nov 03 03:25:46.715202 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:46.715168 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b96c95569-z2wqs" event={"ID":"55406f41-e426-43a4-b621-7a38b12fe150","Type":"ContainerStarted","Data":"fc5c6c077d689b262c7686ec927f4dd775eb98beecb83c8d665ef933799f9bfe"} Nov 03 03:25:46.715649 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:46.715237 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b96c95569-z2wqs" event={"ID":"55406f41-e426-43a4-b621-7a38b12fe150","Type":"ContainerStarted","Data":"8f006474e7b750ef4ab887633333299bb895d60e3e81cce89a35a6392909be76"} Nov 03 03:25:46.715649 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:46.715357 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b96c95569-z2wqs" Nov 03 03:25:46.830845 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:46.830799 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b96c95569-z2wqs" podStartSLOduration=1.83078685 podStartE2EDuration="1.83078685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:46.830147192 +0000 UTC m=+5628.514256347" watchObservedRunningTime="2026-11-03 03:25:46.83078685 +0000 UTC m=+5628.514896004" Nov 03 03:25:57.722441 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:57.722410 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b96c95569-z2wqs" Nov 03 03:25:57.889438 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:57.889402 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55c996f8f7-zx4vc"] Nov 03 03:25:58.250780 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:25:58.250752 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-595669665d-rpgr6"] Nov 03 03:26:00.895709 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:00.895661 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:26:01.343374 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:01.343263 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:26:02.890275 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:02.890212 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55c996f8f7-zx4vc" podUID="af6b24f1-1697-4515-9472-16dac5d82805" containerName="payload-processing" containerID="cri-o://9fba5b81f0914614dbcc970ff5f803659c49d159ca84889a87b65a6fe76e0135" gracePeriod=25 Nov 03 03:26:03.142236 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.142154 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55c996f8f7-zx4vc" Nov 03 03:26:03.251027 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.250986 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-595669665d-rpgr6" podUID="ef792ca8-5cab-4b3a-8417-6708e36bff96" containerName="payload-pre-processing" containerID="cri-o://70efc4d16dbb53b3c76de1da2bd8cb8f991240748940fcd5b996af44cad2f8c0" gracePeriod=25 Nov 03 03:26:03.275715 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.275688 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af6b24f1-1697-4515-9472-16dac5d82805-kube-api-access-mck45\" (UniqueName: \"kubernetes.io/projected/af6b24f1-1697-4515-9472-16dac5d82805-kube-api-access-mck45\") pod \"af6b24f1-1697-4515-9472-16dac5d82805\" (UID: \"af6b24f1-1697-4515-9472-16dac5d82805\") " Nov 03 03:26:03.275871 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.275857 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af6b24f1-1697-4515-9472-16dac5d82805-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af6b24f1-1697-4515-9472-16dac5d82805-plugins-config-volume\") pod \"af6b24f1-1697-4515-9472-16dac5d82805\" (UID: \"af6b24f1-1697-4515-9472-16dac5d82805\") " Nov 03 03:26:03.276216 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.276172 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af6b24f1-1697-4515-9472-16dac5d82805-plugins-config-volume" pod "af6b24f1-1697-4515-9472-16dac5d82805" (UID: "af6b24f1-1697-4515-9472-16dac5d82805"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:26:03.277691 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.277672 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6b24f1-1697-4515-9472-16dac5d82805-kube-api-access-mck45" pod "af6b24f1-1697-4515-9472-16dac5d82805" (UID: "af6b24f1-1697-4515-9472-16dac5d82805"). InnerVolumeSpecName "kube-api-access-mck45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:26:03.377613 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.377584 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af6b24f1-1697-4515-9472-16dac5d82805-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:26:03.377717 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.377619 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mck45\" (UniqueName: \"kubernetes.io/projected/af6b24f1-1697-4515-9472-16dac5d82805-kube-api-access-mck45\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:26:03.491421 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.491396 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-595669665d-rpgr6" Nov 03 03:26:03.680751 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.680671 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef792ca8-5cab-4b3a-8417-6708e36bff96-kube-api-access-4fjf4\" (UniqueName: \"kubernetes.io/projected/ef792ca8-5cab-4b3a-8417-6708e36bff96-kube-api-access-4fjf4\") pod \"ef792ca8-5cab-4b3a-8417-6708e36bff96\" (UID: \"ef792ca8-5cab-4b3a-8417-6708e36bff96\") " Nov 03 03:26:03.680986 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.680819 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef792ca8-5cab-4b3a-8417-6708e36bff96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef792ca8-5cab-4b3a-8417-6708e36bff96-plugins-config-volume\") pod \"ef792ca8-5cab-4b3a-8417-6708e36bff96\" (UID: \"ef792ca8-5cab-4b3a-8417-6708e36bff96\") " Nov 03 03:26:03.681164 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.681142 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef792ca8-5cab-4b3a-8417-6708e36bff96-plugins-config-volume" pod "ef792ca8-5cab-4b3a-8417-6708e36bff96" (UID: "ef792ca8-5cab-4b3a-8417-6708e36bff96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:26:03.682714 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.682694 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef792ca8-5cab-4b3a-8417-6708e36bff96-kube-api-access-4fjf4" pod "ef792ca8-5cab-4b3a-8417-6708e36bff96" (UID: "ef792ca8-5cab-4b3a-8417-6708e36bff96"). InnerVolumeSpecName "kube-api-access-4fjf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:26:03.782466 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.782436 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fjf4\" (UniqueName: \"kubernetes.io/projected/ef792ca8-5cab-4b3a-8417-6708e36bff96-kube-api-access-4fjf4\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:26:03.782466 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.782460 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef792ca8-5cab-4b3a-8417-6708e36bff96-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:26:03.805223 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.805182 5463 generic.go:356] "Generic (PLEG): container finished" podID="af6b24f1-1697-4515-9472-16dac5d82805" containerID="9fba5b81f0914614dbcc970ff5f803659c49d159ca84889a87b65a6fe76e0135" exitCode=0 Nov 03 03:26:03.805372 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.805222 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55c996f8f7-zx4vc" event={"ID":"af6b24f1-1697-4515-9472-16dac5d82805","Type":"ContainerDied","Data":"9fba5b81f0914614dbcc970ff5f803659c49d159ca84889a87b65a6fe76e0135"} Nov 03 03:26:03.805372 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.805273 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55c996f8f7-zx4vc" Nov 03 03:26:03.805372 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.805286 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55c996f8f7-zx4vc" event={"ID":"af6b24f1-1697-4515-9472-16dac5d82805","Type":"ContainerDied","Data":"13a994a718bfde39110c1cdbb6e38897c2d22d3315664e87f591e1fb531edd33"} Nov 03 03:26:03.805372 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.805318 5463 scope.go:122] "RemoveContainer" containerID="9fba5b81f0914614dbcc970ff5f803659c49d159ca84889a87b65a6fe76e0135" Nov 03 03:26:03.806712 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.806693 5463 generic.go:356] "Generic (PLEG): container finished" podID="ef792ca8-5cab-4b3a-8417-6708e36bff96" containerID="70efc4d16dbb53b3c76de1da2bd8cb8f991240748940fcd5b996af44cad2f8c0" exitCode=0 Nov 03 03:26:03.806815 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.806717 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-595669665d-rpgr6" event={"ID":"ef792ca8-5cab-4b3a-8417-6708e36bff96","Type":"ContainerDied","Data":"70efc4d16dbb53b3c76de1da2bd8cb8f991240748940fcd5b996af44cad2f8c0"} Nov 03 03:26:03.806815 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.806735 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-595669665d-rpgr6" Nov 03 03:26:03.806815 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.806767 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-595669665d-rpgr6" event={"ID":"ef792ca8-5cab-4b3a-8417-6708e36bff96","Type":"ContainerDied","Data":"da5020cea7e647f6b8ff00e6994172b11227a40674911c95981e4730bb3039c3"} Nov 03 03:26:03.814308 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.814290 5463 scope.go:122] "RemoveContainer" containerID="9fba5b81f0914614dbcc970ff5f803659c49d159ca84889a87b65a6fe76e0135" Nov 03 03:26:03.814571 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:26:03.814552 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fba5b81f0914614dbcc970ff5f803659c49d159ca84889a87b65a6fe76e0135\": container with ID starting with 9fba5b81f0914614dbcc970ff5f803659c49d159ca84889a87b65a6fe76e0135 not found: ID does not exist" containerID="9fba5b81f0914614dbcc970ff5f803659c49d159ca84889a87b65a6fe76e0135" Nov 03 03:26:03.814652 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.814577 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fba5b81f0914614dbcc970ff5f803659c49d159ca84889a87b65a6fe76e0135"} err="failed to get container status \"9fba5b81f0914614dbcc970ff5f803659c49d159ca84889a87b65a6fe76e0135\": rpc error: code = NotFound desc = could not find container \"9fba5b81f0914614dbcc970ff5f803659c49d159ca84889a87b65a6fe76e0135\": container with ID starting with 9fba5b81f0914614dbcc970ff5f803659c49d159ca84889a87b65a6fe76e0135 not found: ID does not exist" Nov 03 03:26:03.814652 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.814592 5463 scope.go:122] "RemoveContainer" containerID="70efc4d16dbb53b3c76de1da2bd8cb8f991240748940fcd5b996af44cad2f8c0" Nov 03 03:26:03.823312 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.823294 5463 scope.go:122] "RemoveContainer" containerID="70efc4d16dbb53b3c76de1da2bd8cb8f991240748940fcd5b996af44cad2f8c0" Nov 03 03:26:03.823567 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:26:03.823548 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70efc4d16dbb53b3c76de1da2bd8cb8f991240748940fcd5b996af44cad2f8c0\": container with ID starting with 70efc4d16dbb53b3c76de1da2bd8cb8f991240748940fcd5b996af44cad2f8c0 not found: ID does not exist" containerID="70efc4d16dbb53b3c76de1da2bd8cb8f991240748940fcd5b996af44cad2f8c0" Nov 03 03:26:03.823635 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.823578 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70efc4d16dbb53b3c76de1da2bd8cb8f991240748940fcd5b996af44cad2f8c0"} err="failed to get container status \"70efc4d16dbb53b3c76de1da2bd8cb8f991240748940fcd5b996af44cad2f8c0\": rpc error: code = NotFound desc = could not find container \"70efc4d16dbb53b3c76de1da2bd8cb8f991240748940fcd5b996af44cad2f8c0\": container with ID starting with 70efc4d16dbb53b3c76de1da2bd8cb8f991240748940fcd5b996af44cad2f8c0 not found: ID does not exist" Nov 03 03:26:03.972315 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.972240 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55c996f8f7-zx4vc"] Nov 03 03:26:03.976414 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:03.976387 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55c996f8f7-zx4vc"] Nov 03 03:26:04.059665 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:04.059636 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:26:04.090776 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:04.090739 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-595669665d-rpgr6"] Nov 03 03:26:04.094202 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:04.094164 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-595669665d-rpgr6"] Nov 03 03:26:04.441430 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:04.441402 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af6b24f1-1697-4515-9472-16dac5d82805" path="/var/lib/kubelet/pods/af6b24f1-1697-4515-9472-16dac5d82805/volumes" Nov 03 03:26:04.441788 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:04.441775 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef792ca8-5cab-4b3a-8417-6708e36bff96" path="/var/lib/kubelet/pods/ef792ca8-5cab-4b3a-8417-6708e36bff96/volumes" Nov 03 03:26:05.194296 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.194261 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-ztjch"] Nov 03 03:26:05.194809 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.194676 5463 cpu_manager.go:414] "containerMap: removing container" podUID="ef792ca8-5cab-4b3a-8417-6708e36bff96" containerName="payload-pre-processing" Nov 03 03:26:05.194809 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.194687 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef792ca8-5cab-4b3a-8417-6708e36bff96" containerName="payload-pre-processing" Nov 03 03:26:05.194809 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.194703 5463 cpu_manager.go:414] "containerMap: removing container" podUID="af6b24f1-1697-4515-9472-16dac5d82805" containerName="payload-processing" Nov 03 03:26:05.194809 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.194708 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af6b24f1-1697-4515-9472-16dac5d82805" containerName="payload-processing" Nov 03 03:26:05.194809 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.194777 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef792ca8-5cab-4b3a-8417-6708e36bff96" containerName="payload-pre-processing" Nov 03 03:26:05.194809 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.194785 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="af6b24f1-1697-4515-9472-16dac5d82805" containerName="payload-processing" Nov 03 03:26:05.198936 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.198910 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" Nov 03 03:26:05.251264 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.251225 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-ztjch"] Nov 03 03:26:05.298979 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.298948 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 03:26:05.299220 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.298967 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 03:26:05.299220 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.298991 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 03 03:26:05.299451 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.299035 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 03:26:05.299451 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.299329 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-ztjch\" (UID: \"f5993841-ae82-4cf8-9c07-4e72bbba8f4e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" Nov 03 03:26:05.299451 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.299379 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgw7d\" (UniqueName: \"kubernetes.io/projected/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-kube-api-access-lgw7d\") pod \"payload-processing-external-model-5bcd7997-ztjch\" (UID: \"f5993841-ae82-4cf8-9c07-4e72bbba8f4e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" Nov 03 03:26:05.299557 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.299482 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-ztjch\" (UID: \"f5993841-ae82-4cf8-9c07-4e72bbba8f4e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" Nov 03 03:26:05.299557 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.299522 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-ztjch\" (UID: \"f5993841-ae82-4cf8-9c07-4e72bbba8f4e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" Nov 03 03:26:05.302055 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.302034 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-w562g\"" Nov 03 03:26:05.302055 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.302053 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 03:26:05.400674 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.400636 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-ztjch\" (UID: \"f5993841-ae82-4cf8-9c07-4e72bbba8f4e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" Nov 03 03:26:05.400862 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.400712 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-ztjch\" (UID: \"f5993841-ae82-4cf8-9c07-4e72bbba8f4e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" Nov 03 03:26:05.400862 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.400793 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-ztjch\" (UID: \"f5993841-ae82-4cf8-9c07-4e72bbba8f4e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" Nov 03 03:26:05.400862 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.400843 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgw7d\" (UniqueName: \"kubernetes.io/projected/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-kube-api-access-lgw7d\") pod \"payload-processing-external-model-5bcd7997-ztjch\" (UID: \"f5993841-ae82-4cf8-9c07-4e72bbba8f4e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" Nov 03 03:26:05.401523 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.401491 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-ztjch\" (UID: \"f5993841-ae82-4cf8-9c07-4e72bbba8f4e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" Nov 03 03:26:05.401684 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.401563 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-ztjch\" (UID: \"f5993841-ae82-4cf8-9c07-4e72bbba8f4e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" Nov 03 03:26:05.403360 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.403333 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-ztjch\" (UID: \"f5993841-ae82-4cf8-9c07-4e72bbba8f4e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" Nov 03 03:26:05.458319 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.458243 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgw7d\" (UniqueName: \"kubernetes.io/projected/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-kube-api-access-lgw7d\") pod \"payload-processing-external-model-5bcd7997-ztjch\" (UID: \"f5993841-ae82-4cf8-9c07-4e72bbba8f4e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" Nov 03 03:26:05.512527 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.512495 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" Nov 03 03:26:05.886150 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:26:05.886117 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5993841_ae82_4cf8_9c07_4e72bbba8f4e.slice/crio-b29209f5b92ec2550e203e3ecb4bc2af43a08491253d90d9c52c1264f0a371bc WatchSource:0}: Error finding container b29209f5b92ec2550e203e3ecb4bc2af43a08491253d90d9c52c1264f0a371bc: Status 404 returned error can't find the container with id b29209f5b92ec2550e203e3ecb4bc2af43a08491253d90d9c52c1264f0a371bc Nov 03 03:26:05.888589 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.888564 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-ztjch"] Nov 03 03:26:05.948985 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:05.948958 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:26:06.095908 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:06.095870 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:26:06.838675 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:06.838619 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" event={"ID":"f5993841-ae82-4cf8-9c07-4e72bbba8f4e","Type":"ContainerStarted","Data":"3ab560f58401509ea2d5c2c3e5b42eabef2ffcb5da6a200d601f6ce3c833605d"} Nov 03 03:26:06.839245 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:06.838742 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" Nov 03 03:26:06.839245 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:06.838770 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" event={"ID":"f5993841-ae82-4cf8-9c07-4e72bbba8f4e","Type":"ContainerStarted","Data":"b29209f5b92ec2550e203e3ecb4bc2af43a08491253d90d9c52c1264f0a371bc"} Nov 03 03:26:13.005558 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:26:13.005520 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:26:17.846944 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:17.846915 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" Nov 03 03:26:17.961851 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:17.961802 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" podStartSLOduration=12.961788636 podStartE2EDuration="12.961788636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:26:06.958021505 +0000 UTC m=+5648.642130661" watchObservedRunningTime="2026-11-03 03:26:17.961788636 +0000 UTC m=+5659.645897822" Nov 03 03:26:27.382035 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:27.381992 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:26:33.979127 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:33.979082 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:26:43.005642 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:26:43.005563 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:26:47.049539 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:47.049509 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg"] Nov 03 03:26:47.099659 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:47.099629 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0ccd66-hgtxg"] Nov 03 03:26:48.442997 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:26:48.442961 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d952ddac-f153-414e-90f8-31e20b03e42c" path="/var/lib/kubelet/pods/d952ddac-f153-414e-90f8-31e20b03e42c/volumes" Nov 03 03:27:03.441827 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:27:03.441795 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 03:27:03.452477 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:27:03.452451 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 03:27:13.005770 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:27:13.005733 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:27:34.326531 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:27:34.326491 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:27:43.006286 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:27:43.006252 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:27:59.463563 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:27:59.463523 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-ztjch"] Nov 03 03:27:59.593686 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:27:59.593652 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:27:59.747089 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:27:59.746942 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:28:00.151471 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:00.151431 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:28:04.467330 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:04.467255 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" podUID="f5993841-ae82-4cf8-9c07-4e72bbba8f4e" containerName="payload-processing" containerID="cri-o://3ab560f58401509ea2d5c2c3e5b42eabef2ffcb5da6a200d601f6ce3c833605d" gracePeriod=25 Nov 03 03:28:04.760901 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:04.760880 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" Nov 03 03:28:04.848056 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:04.848025 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-routing-overlay-volume\") pod \"f5993841-ae82-4cf8-9c07-4e72bbba8f4e\" (UID: \"f5993841-ae82-4cf8-9c07-4e72bbba8f4e\") " Nov 03 03:28:04.848237 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:04.848083 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-provider-credentials\") pod \"f5993841-ae82-4cf8-9c07-4e72bbba8f4e\" (UID: \"f5993841-ae82-4cf8-9c07-4e72bbba8f4e\") " Nov 03 03:28:04.848237 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:04.848134 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-kube-api-access-lgw7d\" (UniqueName: \"kubernetes.io/projected/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-kube-api-access-lgw7d\") pod \"f5993841-ae82-4cf8-9c07-4e72bbba8f4e\" (UID: \"f5993841-ae82-4cf8-9c07-4e72bbba8f4e\") " Nov 03 03:28:04.848237 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:04.848180 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-plugins-config-volume\") pod \"f5993841-ae82-4cf8-9c07-4e72bbba8f4e\" (UID: \"f5993841-ae82-4cf8-9c07-4e72bbba8f4e\") " Nov 03 03:28:04.848471 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:04.848378 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-routing-overlay-volume" pod "f5993841-ae82-4cf8-9c07-4e72bbba8f4e" (UID: "f5993841-ae82-4cf8-9c07-4e72bbba8f4e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:28:04.848601 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:04.848584 5463 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-routing-overlay-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:28:04.848672 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:04.848612 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-plugins-config-volume" pod "f5993841-ae82-4cf8-9c07-4e72bbba8f4e" (UID: "f5993841-ae82-4cf8-9c07-4e72bbba8f4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:28:04.850332 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:04.850309 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-kube-api-access-lgw7d" pod "f5993841-ae82-4cf8-9c07-4e72bbba8f4e" (UID: "f5993841-ae82-4cf8-9c07-4e72bbba8f4e"). InnerVolumeSpecName "kube-api-access-lgw7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:28:04.850421 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:04.850376 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-provider-credentials" pod "f5993841-ae82-4cf8-9c07-4e72bbba8f4e" (UID: "f5993841-ae82-4cf8-9c07-4e72bbba8f4e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:28:04.949353 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:04.949318 5463 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-provider-credentials\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:28:04.949353 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:04.949349 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgw7d\" (UniqueName: \"kubernetes.io/projected/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-kube-api-access-lgw7d\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:28:04.949560 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:04.949365 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5993841-ae82-4cf8-9c07-4e72bbba8f4e-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:28:05.498851 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:05.498815 5463 generic.go:356] "Generic (PLEG): container finished" podID="f5993841-ae82-4cf8-9c07-4e72bbba8f4e" containerID="3ab560f58401509ea2d5c2c3e5b42eabef2ffcb5da6a200d601f6ce3c833605d" exitCode=0 Nov 03 03:28:05.499410 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:05.498861 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" Nov 03 03:28:05.499410 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:05.498896 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" event={"ID":"f5993841-ae82-4cf8-9c07-4e72bbba8f4e","Type":"ContainerDied","Data":"3ab560f58401509ea2d5c2c3e5b42eabef2ffcb5da6a200d601f6ce3c833605d"} Nov 03 03:28:05.499410 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:05.498955 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ztjch" event={"ID":"f5993841-ae82-4cf8-9c07-4e72bbba8f4e","Type":"ContainerDied","Data":"b29209f5b92ec2550e203e3ecb4bc2af43a08491253d90d9c52c1264f0a371bc"} Nov 03 03:28:05.499410 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:05.498983 5463 scope.go:122] "RemoveContainer" containerID="3ab560f58401509ea2d5c2c3e5b42eabef2ffcb5da6a200d601f6ce3c833605d" Nov 03 03:28:05.509221 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:05.509178 5463 scope.go:122] "RemoveContainer" containerID="3ab560f58401509ea2d5c2c3e5b42eabef2ffcb5da6a200d601f6ce3c833605d" Nov 03 03:28:05.509525 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:28:05.509505 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ab560f58401509ea2d5c2c3e5b42eabef2ffcb5da6a200d601f6ce3c833605d\": container with ID starting with 3ab560f58401509ea2d5c2c3e5b42eabef2ffcb5da6a200d601f6ce3c833605d not found: ID does not exist" containerID="3ab560f58401509ea2d5c2c3e5b42eabef2ffcb5da6a200d601f6ce3c833605d" Nov 03 03:28:05.509598 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:05.509535 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab560f58401509ea2d5c2c3e5b42eabef2ffcb5da6a200d601f6ce3c833605d"} err="failed to get container status \"3ab560f58401509ea2d5c2c3e5b42eabef2ffcb5da6a200d601f6ce3c833605d\": rpc error: code = NotFound desc = could not find container \"3ab560f58401509ea2d5c2c3e5b42eabef2ffcb5da6a200d601f6ce3c833605d\": container with ID starting with 3ab560f58401509ea2d5c2c3e5b42eabef2ffcb5da6a200d601f6ce3c833605d not found: ID does not exist" Nov 03 03:28:05.697166 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:05.697138 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-ztjch"] Nov 03 03:28:05.700688 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:05.700666 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-ztjch"] Nov 03 03:28:06.311254 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:06.311221 5463 scope.go:122] "RemoveContainer" containerID="c60f1eea75a0da4cd4982756557c740b610b205f84916a17334848a1676d2250" Nov 03 03:28:06.441005 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:06.440968 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5993841-ae82-4cf8-9c07-4e72bbba8f4e" path="/var/lib/kubelet/pods/f5993841-ae82-4cf8-9c07-4e72bbba8f4e/volumes" Nov 03 03:28:13.008363 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:28:13.008272 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:28:18.417585 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:18.417546 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t"] Nov 03 03:28:18.418095 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:18.418026 5463 cpu_manager.go:414] "containerMap: removing container" podUID="f5993841-ae82-4cf8-9c07-4e72bbba8f4e" containerName="payload-processing" Nov 03 03:28:18.418095 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:18.418038 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5993841-ae82-4cf8-9c07-4e72bbba8f4e" containerName="payload-processing" Nov 03 03:28:18.418245 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:18.418136 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5993841-ae82-4cf8-9c07-4e72bbba8f4e" containerName="payload-processing" Nov 03 03:28:18.420433 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:18.420409 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t" Nov 03 03:28:18.474277 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:18.474242 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t"] Nov 03 03:28:18.498674 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:18.498648 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf1ea68-5916-4161-9461-096c8a6dbec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-shm5t\" (UID: \"dbf1ea68-5916-4161-9461-096c8a6dbec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t" Nov 03 03:28:18.498818 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:18.498701 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf1ea68-5916-4161-9461-096c8a6dbec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-shm5t\" (UID: \"dbf1ea68-5916-4161-9461-096c8a6dbec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t" Nov 03 03:28:18.498818 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:18.498726 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2r6h\" (UniqueName: \"kubernetes.io/projected/dbf1ea68-5916-4161-9461-096c8a6dbec8-kube-api-access-h2r6h\") pod \"maas-api-revoke-sub-c1e271a9-shm5t\" (UID: \"dbf1ea68-5916-4161-9461-096c8a6dbec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t" Nov 03 03:28:18.521718 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:18.521692 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:28:18.521718 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:18.521692 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:28:18.521953 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:18.521721 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:28:18.523573 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:18.523553 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:28:18.599845 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:18.599804 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf1ea68-5916-4161-9461-096c8a6dbec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-shm5t\" (UID: \"dbf1ea68-5916-4161-9461-096c8a6dbec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t" Nov 03 03:28:18.600057 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:18.599855 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2r6h\" (UniqueName: \"kubernetes.io/projected/dbf1ea68-5916-4161-9461-096c8a6dbec8-kube-api-access-h2r6h\") pod \"maas-api-revoke-sub-c1e271a9-shm5t\" (UID: \"dbf1ea68-5916-4161-9461-096c8a6dbec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t" Nov 03 03:28:18.600057 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:18.600005 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf1ea68-5916-4161-9461-096c8a6dbec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-shm5t\" (UID: \"dbf1ea68-5916-4161-9461-096c8a6dbec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t" Nov 03 03:28:18.601172 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:18.601147 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf1ea68-5916-4161-9461-096c8a6dbec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-shm5t\" (UID: \"dbf1ea68-5916-4161-9461-096c8a6dbec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t" Nov 03 03:28:18.602715 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:18.602689 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf1ea68-5916-4161-9461-096c8a6dbec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-shm5t\" (UID: \"dbf1ea68-5916-4161-9461-096c8a6dbec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t" Nov 03 03:28:18.657952 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:18.657919 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2r6h\" (UniqueName: \"kubernetes.io/projected/dbf1ea68-5916-4161-9461-096c8a6dbec8-kube-api-access-h2r6h\") pod \"maas-api-revoke-sub-c1e271a9-shm5t\" (UID: \"dbf1ea68-5916-4161-9461-096c8a6dbec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t" Nov 03 03:28:18.738034 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:18.737960 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t" Nov 03 03:28:19.104820 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:28:19.104785 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf1ea68_5916_4161_9461_096c8a6dbec8.slice/crio-614a7f3017cd6fd719e6de5abdd78c58e39b14a094faf02e4369900a23d83185 WatchSource:0}: Error finding container 614a7f3017cd6fd719e6de5abdd78c58e39b14a094faf02e4369900a23d83185: Status 404 returned error can't find the container with id 614a7f3017cd6fd719e6de5abdd78c58e39b14a094faf02e4369900a23d83185 Nov 03 03:28:19.106747 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:19.106721 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t"] Nov 03 03:28:19.621515 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:19.621425 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t" event={"ID":"dbf1ea68-5916-4161-9461-096c8a6dbec8","Type":"ContainerStarted","Data":"8b75df93541acecad5b41ddcf6621fcb10f9fd866038d35c453b98c3bf7786dc"} Nov 03 03:28:19.621515 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:19.621489 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t" event={"ID":"dbf1ea68-5916-4161-9461-096c8a6dbec8","Type":"ContainerStarted","Data":"614a7f3017cd6fd719e6de5abdd78c58e39b14a094faf02e4369900a23d83185"} Nov 03 03:28:19.736559 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:19.736505 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t" podStartSLOduration=1.7364916419999998 podStartE2EDuration="1.736491642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:28:19.735186222 +0000 UTC m=+5781.419295378" watchObservedRunningTime="2026-11-03 03:28:19.736491642 +0000 UTC m=+5781.420600797" Nov 03 03:28:20.627988 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:20.627950 5463 generic.go:356] "Generic (PLEG): container finished" podID="dbf1ea68-5916-4161-9461-096c8a6dbec8" containerID="8b75df93541acecad5b41ddcf6621fcb10f9fd866038d35c453b98c3bf7786dc" exitCode=0 Nov 03 03:28:20.628406 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:20.628035 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t" event={"ID":"dbf1ea68-5916-4161-9461-096c8a6dbec8","Type":"ContainerDied","Data":"8b75df93541acecad5b41ddcf6621fcb10f9fd866038d35c453b98c3bf7786dc"} Nov 03 03:28:21.772485 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:21.772459 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t" Nov 03 03:28:21.829684 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:21.829660 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbf1ea68-5916-4161-9461-096c8a6dbec8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf1ea68-5916-4161-9461-096c8a6dbec8-maas-api-xks-ca\") pod \"dbf1ea68-5916-4161-9461-096c8a6dbec8\" (UID: \"dbf1ea68-5916-4161-9461-096c8a6dbec8\") " Nov 03 03:28:21.829804 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:21.829730 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbf1ea68-5916-4161-9461-096c8a6dbec8-kube-api-access-h2r6h\" (UniqueName: \"kubernetes.io/projected/dbf1ea68-5916-4161-9461-096c8a6dbec8-kube-api-access-h2r6h\") pod \"dbf1ea68-5916-4161-9461-096c8a6dbec8\" (UID: \"dbf1ea68-5916-4161-9461-096c8a6dbec8\") " Nov 03 03:28:21.829804 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:21.829778 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbf1ea68-5916-4161-9461-096c8a6dbec8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf1ea68-5916-4161-9461-096c8a6dbec8-maas-api-openshift-ca\") pod \"dbf1ea68-5916-4161-9461-096c8a6dbec8\" (UID: \"dbf1ea68-5916-4161-9461-096c8a6dbec8\") " Nov 03 03:28:21.830175 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:21.830146 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf1ea68-5916-4161-9461-096c8a6dbec8-maas-api-openshift-ca" pod "dbf1ea68-5916-4161-9461-096c8a6dbec8" (UID: "dbf1ea68-5916-4161-9461-096c8a6dbec8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:28:21.831725 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:21.831706 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbf1ea68-5916-4161-9461-096c8a6dbec8-maas-api-xks-ca" pod "dbf1ea68-5916-4161-9461-096c8a6dbec8" (UID: "dbf1ea68-5916-4161-9461-096c8a6dbec8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:28:21.831803 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:21.831772 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf1ea68-5916-4161-9461-096c8a6dbec8-kube-api-access-h2r6h" pod "dbf1ea68-5916-4161-9461-096c8a6dbec8" (UID: "dbf1ea68-5916-4161-9461-096c8a6dbec8"). InnerVolumeSpecName "kube-api-access-h2r6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:28:21.930944 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:21.930912 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf1ea68-5916-4161-9461-096c8a6dbec8-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:28:21.930944 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:21.930942 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2r6h\" (UniqueName: \"kubernetes.io/projected/dbf1ea68-5916-4161-9461-096c8a6dbec8-kube-api-access-h2r6h\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:28:21.931154 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:21.930953 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf1ea68-5916-4161-9461-096c8a6dbec8-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:28:22.640104 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:22.640067 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t" event={"ID":"dbf1ea68-5916-4161-9461-096c8a6dbec8","Type":"ContainerDied","Data":"614a7f3017cd6fd719e6de5abdd78c58e39b14a094faf02e4369900a23d83185"} Nov 03 03:28:22.640302 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:22.640130 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="614a7f3017cd6fd719e6de5abdd78c58e39b14a094faf02e4369900a23d83185" Nov 03 03:28:22.640302 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:22.640082 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t" Nov 03 03:28:34.700628 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:34.700591 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:28:42.688587 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:42.688548 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc"] Nov 03 03:28:42.689056 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:42.689020 5463 cpu_manager.go:414] "containerMap: removing container" podUID="dbf1ea68-5916-4161-9461-096c8a6dbec8" containerName="revoke-keys" Nov 03 03:28:42.689056 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:42.689036 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbf1ea68-5916-4161-9461-096c8a6dbec8" containerName="revoke-keys" Nov 03 03:28:42.689139 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:42.689114 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbf1ea68-5916-4161-9461-096c8a6dbec8" containerName="revoke-keys" Nov 03 03:28:42.692605 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:42.692582 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc" Nov 03 03:28:42.744863 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:42.744831 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc"] Nov 03 03:28:42.756696 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:42.756660 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p444b\" (UniqueName: \"kubernetes.io/projected/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-kube-api-access-p444b\") pod \"maas-api-revoke-sub-2e1f5b5f-6zhbc\" (UID: \"cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc" Nov 03 03:28:42.756900 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:42.756871 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-6zhbc\" (UID: \"cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc" Nov 03 03:28:42.757008 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:42.756931 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-6zhbc\" (UID: \"cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc" Nov 03 03:28:42.792701 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:42.792663 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:28:42.792862 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:42.792705 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:28:42.792862 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:42.792721 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:28:42.795563 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:42.795546 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:28:42.857810 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:42.857779 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-6zhbc\" (UID: \"cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc" Nov 03 03:28:42.857960 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:42.857829 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p444b\" (UniqueName: \"kubernetes.io/projected/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-kube-api-access-p444b\") pod \"maas-api-revoke-sub-2e1f5b5f-6zhbc\" (UID: \"cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc" Nov 03 03:28:42.858011 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:42.857961 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-6zhbc\" (UID: \"cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc" Nov 03 03:28:42.858600 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:42.858580 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-6zhbc\" (UID: \"cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc" Nov 03 03:28:42.860429 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:42.860402 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-6zhbc\" (UID: \"cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc" Nov 03 03:28:42.915999 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:42.915964 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p444b\" (UniqueName: \"kubernetes.io/projected/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-kube-api-access-p444b\") pod \"maas-api-revoke-sub-2e1f5b5f-6zhbc\" (UID: \"cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc" Nov 03 03:28:43.006110 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:28:43.006030 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:28:43.006110 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:43.006061 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc" Nov 03 03:28:43.373824 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:28:43.373791 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7d0d6a_8eb8_4aa4_a4bb_8b7fdc8194bc.slice/crio-5691f988d7c0c5b846fbcf7a5033e54213883e8ae7b184094ce0083b2ba512d0 WatchSource:0}: Error finding container 5691f988d7c0c5b846fbcf7a5033e54213883e8ae7b184094ce0083b2ba512d0: Status 404 returned error can't find the container with id 5691f988d7c0c5b846fbcf7a5033e54213883e8ae7b184094ce0083b2ba512d0 Nov 03 03:28:43.376328 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:43.376296 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc"] Nov 03 03:28:43.760338 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:43.760300 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc" event={"ID":"cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc","Type":"ContainerStarted","Data":"76ff7011e25eae404fc0f2e50232ab56ccc4020095c38e7bf531ca4151cf5a73"} Nov 03 03:28:43.760786 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:43.760359 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc" event={"ID":"cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc","Type":"ContainerStarted","Data":"5691f988d7c0c5b846fbcf7a5033e54213883e8ae7b184094ce0083b2ba512d0"} Nov 03 03:28:43.874436 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:43.874392 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc" podStartSLOduration=1.874378396 podStartE2EDuration="1.874378396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:28:43.873792969 +0000 UTC m=+5805.557902135" watchObservedRunningTime="2026-11-03 03:28:43.874378396 +0000 UTC m=+5805.558487550" Nov 03 03:28:44.767092 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:44.767061 5463 generic.go:356] "Generic (PLEG): container finished" podID="cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc" containerID="76ff7011e25eae404fc0f2e50232ab56ccc4020095c38e7bf531ca4151cf5a73" exitCode=0 Nov 03 03:28:44.767520 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:44.767155 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc" event={"ID":"cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc","Type":"ContainerDied","Data":"76ff7011e25eae404fc0f2e50232ab56ccc4020095c38e7bf531ca4151cf5a73"} Nov 03 03:28:45.921011 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:45.920988 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc" Nov 03 03:28:45.989020 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:45.988990 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-maas-api-openshift-ca\") pod \"cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc\" (UID: \"cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc\") " Nov 03 03:28:45.989177 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:45.989107 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-maas-api-xks-ca\") pod \"cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc\" (UID: \"cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc\") " Nov 03 03:28:45.989177 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:45.989128 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-kube-api-access-p444b\" (UniqueName: \"kubernetes.io/projected/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-kube-api-access-p444b\") pod \"cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc\" (UID: \"cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc\") " Nov 03 03:28:45.989454 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:45.989425 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-maas-api-openshift-ca" pod "cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc" (UID: "cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:28:45.991265 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:45.991231 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-maas-api-xks-ca" pod "cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc" (UID: "cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:28:45.991386 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:45.991369 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-kube-api-access-p444b" pod "cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc" (UID: "cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc"). InnerVolumeSpecName "kube-api-access-p444b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:28:46.090594 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:46.090522 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:28:46.090594 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:46.090550 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p444b\" (UniqueName: \"kubernetes.io/projected/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-kube-api-access-p444b\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:28:46.090594 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:46.090561 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:28:46.781048 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:46.780963 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc" Nov 03 03:28:46.781048 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:46.780965 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc" event={"ID":"cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc","Type":"ContainerDied","Data":"5691f988d7c0c5b846fbcf7a5033e54213883e8ae7b184094ce0083b2ba512d0"} Nov 03 03:28:46.781392 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:46.781086 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5691f988d7c0c5b846fbcf7a5033e54213883e8ae7b184094ce0083b2ba512d0" Nov 03 03:28:55.111778 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.111742 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn"] Nov 03 03:28:55.112295 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.112172 5463 cpu_manager.go:414] "containerMap: removing container" podUID="cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc" containerName="revoke-keys" Nov 03 03:28:55.112295 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.112183 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc" containerName="revoke-keys" Nov 03 03:28:55.112295 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.112260 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc" containerName="revoke-keys" Nov 03 03:28:55.115763 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.115735 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn" Nov 03 03:28:55.169760 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.169726 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn"] Nov 03 03:28:55.178622 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.178590 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b277dbf3-c170-44d4-a1a3-1587623db747-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4djvn\" (UID: \"b277dbf3-c170-44d4-a1a3-1587623db747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn" Nov 03 03:28:55.178755 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.178696 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxb9f\" (UniqueName: \"kubernetes.io/projected/b277dbf3-c170-44d4-a1a3-1587623db747-kube-api-access-zxb9f\") pod \"maas-api-revoke-sub-ec843018-4djvn\" (UID: \"b277dbf3-c170-44d4-a1a3-1587623db747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn" Nov 03 03:28:55.178892 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.178871 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b277dbf3-c170-44d4-a1a3-1587623db747-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4djvn\" (UID: \"b277dbf3-c170-44d4-a1a3-1587623db747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn" Nov 03 03:28:55.216917 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.216890 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:28:55.217065 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.216980 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:28:55.267387 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.267366 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:28:55.267387 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.267366 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:28:55.279921 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.279898 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b277dbf3-c170-44d4-a1a3-1587623db747-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4djvn\" (UID: \"b277dbf3-c170-44d4-a1a3-1587623db747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn" Nov 03 03:28:55.280051 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.279939 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxb9f\" (UniqueName: \"kubernetes.io/projected/b277dbf3-c170-44d4-a1a3-1587623db747-kube-api-access-zxb9f\") pod \"maas-api-revoke-sub-ec843018-4djvn\" (UID: \"b277dbf3-c170-44d4-a1a3-1587623db747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn" Nov 03 03:28:55.280051 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.280028 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b277dbf3-c170-44d4-a1a3-1587623db747-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4djvn\" (UID: \"b277dbf3-c170-44d4-a1a3-1587623db747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn" Nov 03 03:28:55.280669 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.280644 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b277dbf3-c170-44d4-a1a3-1587623db747-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4djvn\" (UID: \"b277dbf3-c170-44d4-a1a3-1587623db747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn" Nov 03 03:28:55.282264 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.282246 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b277dbf3-c170-44d4-a1a3-1587623db747-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4djvn\" (UID: \"b277dbf3-c170-44d4-a1a3-1587623db747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn" Nov 03 03:28:55.338690 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.338660 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxb9f\" (UniqueName: \"kubernetes.io/projected/b277dbf3-c170-44d4-a1a3-1587623db747-kube-api-access-zxb9f\") pod \"maas-api-revoke-sub-ec843018-4djvn\" (UID: \"b277dbf3-c170-44d4-a1a3-1587623db747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn" Nov 03 03:28:55.428774 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.428745 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn" Nov 03 03:28:55.798229 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:28:55.798181 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb277dbf3_c170_44d4_a1a3_1587623db747.slice/crio-eeb4c76e1b1cb25ffee09d92bfd053dda8087dd1baba2f99f5b8a0895c637bfc WatchSource:0}: Error finding container eeb4c76e1b1cb25ffee09d92bfd053dda8087dd1baba2f99f5b8a0895c637bfc: Status 404 returned error can't find the container with id eeb4c76e1b1cb25ffee09d92bfd053dda8087dd1baba2f99f5b8a0895c637bfc Nov 03 03:28:55.800160 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.800137 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn"] Nov 03 03:28:55.830383 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:55.830351 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn" event={"ID":"b277dbf3-c170-44d4-a1a3-1587623db747","Type":"ContainerStarted","Data":"eeb4c76e1b1cb25ffee09d92bfd053dda8087dd1baba2f99f5b8a0895c637bfc"} Nov 03 03:28:56.837779 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:56.837692 5463 generic.go:356] "Generic (PLEG): container finished" podID="b277dbf3-c170-44d4-a1a3-1587623db747" containerID="27c9c8bae79397504e578c610a96f139a7d3f5dc6f262caac226d30c70892f0b" exitCode=0 Nov 03 03:28:56.837779 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:56.837744 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn" event={"ID":"b277dbf3-c170-44d4-a1a3-1587623db747","Type":"ContainerDied","Data":"27c9c8bae79397504e578c610a96f139a7d3f5dc6f262caac226d30c70892f0b"} Nov 03 03:28:57.980942 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:57.980920 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn" Nov 03 03:28:58.106394 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:58.106330 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b277dbf3-c170-44d4-a1a3-1587623db747-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b277dbf3-c170-44d4-a1a3-1587623db747-maas-api-openshift-ca\") pod \"b277dbf3-c170-44d4-a1a3-1587623db747\" (UID: \"b277dbf3-c170-44d4-a1a3-1587623db747\") " Nov 03 03:28:58.106550 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:58.106411 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b277dbf3-c170-44d4-a1a3-1587623db747-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b277dbf3-c170-44d4-a1a3-1587623db747-maas-api-xks-ca\") pod \"b277dbf3-c170-44d4-a1a3-1587623db747\" (UID: \"b277dbf3-c170-44d4-a1a3-1587623db747\") " Nov 03 03:28:58.106550 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:58.106461 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b277dbf3-c170-44d4-a1a3-1587623db747-kube-api-access-zxb9f\" (UniqueName: \"kubernetes.io/projected/b277dbf3-c170-44d4-a1a3-1587623db747-kube-api-access-zxb9f\") pod \"b277dbf3-c170-44d4-a1a3-1587623db747\" (UID: \"b277dbf3-c170-44d4-a1a3-1587623db747\") " Nov 03 03:28:58.106766 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:58.106740 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b277dbf3-c170-44d4-a1a3-1587623db747-maas-api-openshift-ca" pod "b277dbf3-c170-44d4-a1a3-1587623db747" (UID: "b277dbf3-c170-44d4-a1a3-1587623db747"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:28:58.108522 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:58.108499 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b277dbf3-c170-44d4-a1a3-1587623db747-maas-api-xks-ca" pod "b277dbf3-c170-44d4-a1a3-1587623db747" (UID: "b277dbf3-c170-44d4-a1a3-1587623db747"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:28:58.108630 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:58.108575 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b277dbf3-c170-44d4-a1a3-1587623db747-kube-api-access-zxb9f" pod "b277dbf3-c170-44d4-a1a3-1587623db747" (UID: "b277dbf3-c170-44d4-a1a3-1587623db747"). InnerVolumeSpecName "kube-api-access-zxb9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:28:58.207496 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:58.207473 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b277dbf3-c170-44d4-a1a3-1587623db747-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:28:58.207496 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:58.207496 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b277dbf3-c170-44d4-a1a3-1587623db747-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:28:58.207644 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:58.207506 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxb9f\" (UniqueName: \"kubernetes.io/projected/b277dbf3-c170-44d4-a1a3-1587623db747-kube-api-access-zxb9f\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:28:58.850018 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:58.849923 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn" Nov 03 03:28:58.850177 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:58.849926 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn" event={"ID":"b277dbf3-c170-44d4-a1a3-1587623db747","Type":"ContainerDied","Data":"eeb4c76e1b1cb25ffee09d92bfd053dda8087dd1baba2f99f5b8a0895c637bfc"} Nov 03 03:28:58.850177 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:28:58.850096 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeb4c76e1b1cb25ffee09d92bfd053dda8087dd1baba2f99f5b8a0895c637bfc" Nov 03 03:29:13.015118 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:29:13.015075 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:29:16.047093 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:16.047063 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6"] Nov 03 03:29:16.097114 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:16.097085 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cj4r6"] Nov 03 03:29:16.442211 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:16.442145 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a332fc1c-f31b-4318-897d-0f2f8b14e889" path="/var/lib/kubelet/pods/a332fc1c-f31b-4318-897d-0f2f8b14e889/volumes" Nov 03 03:29:30.357263 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:30.357228 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l"] Nov 03 03:29:30.357653 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:30.357633 5463 cpu_manager.go:414] "containerMap: removing container" podUID="b277dbf3-c170-44d4-a1a3-1587623db747" containerName="revoke-keys" Nov 03 03:29:30.357653 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:30.357644 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b277dbf3-c170-44d4-a1a3-1587623db747" containerName="revoke-keys" Nov 03 03:29:30.357739 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:30.357728 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="b277dbf3-c170-44d4-a1a3-1587623db747" containerName="revoke-keys" Nov 03 03:29:30.360993 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:30.360976 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l" Nov 03 03:29:30.414905 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:30.414872 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l"] Nov 03 03:29:30.415171 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:30.415145 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:29:30.415335 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:30.415184 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:29:30.447759 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:30.447731 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-77m5l\" (UID: \"70dc5d7a-d0a0-499d-a82c-ddc02cfbe426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l" Nov 03 03:29:30.447905 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:30.447809 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-77m5l\" (UID: \"70dc5d7a-d0a0-499d-a82c-ddc02cfbe426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l" Nov 03 03:29:30.447905 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:30.447828 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4bwc\" (UniqueName: \"kubernetes.io/projected/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-kube-api-access-p4bwc\") pod \"maas-api-revoke-sub-046e1343-77m5l\" (UID: \"70dc5d7a-d0a0-499d-a82c-ddc02cfbe426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l" Nov 03 03:29:30.463208 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:30.463170 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:29:30.463208 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:30.463180 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:29:30.549158 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:30.549131 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-77m5l\" (UID: \"70dc5d7a-d0a0-499d-a82c-ddc02cfbe426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l" Nov 03 03:29:30.549355 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:30.549311 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-77m5l\" (UID: \"70dc5d7a-d0a0-499d-a82c-ddc02cfbe426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l" Nov 03 03:29:30.549355 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:30.549350 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4bwc\" (UniqueName: \"kubernetes.io/projected/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-kube-api-access-p4bwc\") pod \"maas-api-revoke-sub-046e1343-77m5l\" (UID: \"70dc5d7a-d0a0-499d-a82c-ddc02cfbe426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l" Nov 03 03:29:30.549928 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:30.549903 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-77m5l\" (UID: \"70dc5d7a-d0a0-499d-a82c-ddc02cfbe426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l" Nov 03 03:29:30.551538 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:30.551513 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-77m5l\" (UID: \"70dc5d7a-d0a0-499d-a82c-ddc02cfbe426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l" Nov 03 03:29:30.607392 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:30.607338 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4bwc\" (UniqueName: \"kubernetes.io/projected/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-kube-api-access-p4bwc\") pod \"maas-api-revoke-sub-046e1343-77m5l\" (UID: \"70dc5d7a-d0a0-499d-a82c-ddc02cfbe426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l" Nov 03 03:29:30.673624 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:30.673597 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l" Nov 03 03:29:31.041742 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:29:31.041711 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70dc5d7a_d0a0_499d_a82c_ddc02cfbe426.slice/crio-6cc657ba68fec776f1e6d6fde0d703ab1ca2b43bee1254ffffa188a31af6aa5b WatchSource:0}: Error finding container 6cc657ba68fec776f1e6d6fde0d703ab1ca2b43bee1254ffffa188a31af6aa5b: Status 404 returned error can't find the container with id 6cc657ba68fec776f1e6d6fde0d703ab1ca2b43bee1254ffffa188a31af6aa5b Nov 03 03:29:31.043581 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:31.043558 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l"] Nov 03 03:29:31.066966 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:31.066936 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l" event={"ID":"70dc5d7a-d0a0-499d-a82c-ddc02cfbe426","Type":"ContainerStarted","Data":"6cc657ba68fec776f1e6d6fde0d703ab1ca2b43bee1254ffffa188a31af6aa5b"} Nov 03 03:29:32.073456 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:32.073419 5463 generic.go:356] "Generic (PLEG): container finished" podID="70dc5d7a-d0a0-499d-a82c-ddc02cfbe426" containerID="b695e33d0b8fbf52b09c0d89f91b945b537d9e9a70eb161027bfb713264e6b28" exitCode=0 Nov 03 03:29:32.073861 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:32.073472 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l" event={"ID":"70dc5d7a-d0a0-499d-a82c-ddc02cfbe426","Type":"ContainerDied","Data":"b695e33d0b8fbf52b09c0d89f91b945b537d9e9a70eb161027bfb713264e6b28"} Nov 03 03:29:33.212450 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:33.212426 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l" Nov 03 03:29:33.275340 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:33.275302 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-kube-api-access-p4bwc\" (UniqueName: \"kubernetes.io/projected/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-kube-api-access-p4bwc\") pod \"70dc5d7a-d0a0-499d-a82c-ddc02cfbe426\" (UID: \"70dc5d7a-d0a0-499d-a82c-ddc02cfbe426\") " Nov 03 03:29:33.275496 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:33.275391 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-maas-api-xks-ca\") pod \"70dc5d7a-d0a0-499d-a82c-ddc02cfbe426\" (UID: \"70dc5d7a-d0a0-499d-a82c-ddc02cfbe426\") " Nov 03 03:29:33.275496 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:33.275460 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-maas-api-openshift-ca\") pod \"70dc5d7a-d0a0-499d-a82c-ddc02cfbe426\" (UID: \"70dc5d7a-d0a0-499d-a82c-ddc02cfbe426\") " Nov 03 03:29:33.275837 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:33.275810 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-maas-api-openshift-ca" pod "70dc5d7a-d0a0-499d-a82c-ddc02cfbe426" (UID: "70dc5d7a-d0a0-499d-a82c-ddc02cfbe426"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:29:33.277431 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:33.277404 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-kube-api-access-p4bwc" pod "70dc5d7a-d0a0-499d-a82c-ddc02cfbe426" (UID: "70dc5d7a-d0a0-499d-a82c-ddc02cfbe426"). InnerVolumeSpecName "kube-api-access-p4bwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:29:33.277563 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:33.277421 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-maas-api-xks-ca" pod "70dc5d7a-d0a0-499d-a82c-ddc02cfbe426" (UID: "70dc5d7a-d0a0-499d-a82c-ddc02cfbe426"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:29:33.376612 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:33.376542 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:29:33.376612 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:33.376569 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:29:33.376612 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:33.376580 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4bwc\" (UniqueName: \"kubernetes.io/projected/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426-kube-api-access-p4bwc\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:29:34.084764 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:34.084734 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l" Nov 03 03:29:34.084944 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:34.084780 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l" event={"ID":"70dc5d7a-d0a0-499d-a82c-ddc02cfbe426","Type":"ContainerDied","Data":"6cc657ba68fec776f1e6d6fde0d703ab1ca2b43bee1254ffffa188a31af6aa5b"} Nov 03 03:29:34.084944 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:34.084829 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cc657ba68fec776f1e6d6fde0d703ab1ca2b43bee1254ffffa188a31af6aa5b" Nov 03 03:29:34.992658 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:34.992612 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:29:39.046755 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:39.046680 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh"] Nov 03 03:29:39.097230 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:39.097184 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sz4wh"] Nov 03 03:29:40.441500 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:40.441465 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77654092-ca40-429e-a5b9-68ea634c5d48" path="/var/lib/kubelet/pods/77654092-ca40-429e-a5b9-68ea634c5d48/volumes" Nov 03 03:29:41.154777 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.154734 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9"] Nov 03 03:29:41.155348 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.155326 5463 cpu_manager.go:414] "containerMap: removing container" podUID="70dc5d7a-d0a0-499d-a82c-ddc02cfbe426" containerName="revoke-keys" Nov 03 03:29:41.155348 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.155348 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70dc5d7a-d0a0-499d-a82c-ddc02cfbe426" containerName="revoke-keys" Nov 03 03:29:41.155531 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.155446 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="70dc5d7a-d0a0-499d-a82c-ddc02cfbe426" containerName="revoke-keys" Nov 03 03:29:41.158828 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.158807 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9" Nov 03 03:29:41.210447 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.210414 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9"] Nov 03 03:29:41.248556 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.248529 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mptl9\" (UniqueName: \"kubernetes.io/projected/9bd5d230-0bbe-4410-853d-b149c782dbf9-kube-api-access-mptl9\") pod \"maas-api-revoke-sub-d3fba056-k95m9\" (UID: \"9bd5d230-0bbe-4410-853d-b149c782dbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9" Nov 03 03:29:41.248697 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.248593 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd5d230-0bbe-4410-853d-b149c782dbf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-k95m9\" (UID: \"9bd5d230-0bbe-4410-853d-b149c782dbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9" Nov 03 03:29:41.248741 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.248703 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd5d230-0bbe-4410-853d-b149c782dbf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-k95m9\" (UID: \"9bd5d230-0bbe-4410-853d-b149c782dbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9" Nov 03 03:29:41.258112 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.258091 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:29:41.258242 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.258122 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:29:41.263789 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.263770 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:29:41.263789 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.263784 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:29:41.349799 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.349770 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mptl9\" (UniqueName: \"kubernetes.io/projected/9bd5d230-0bbe-4410-853d-b149c782dbf9-kube-api-access-mptl9\") pod \"maas-api-revoke-sub-d3fba056-k95m9\" (UID: \"9bd5d230-0bbe-4410-853d-b149c782dbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9" Nov 03 03:29:41.349967 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.349829 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd5d230-0bbe-4410-853d-b149c782dbf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-k95m9\" (UID: \"9bd5d230-0bbe-4410-853d-b149c782dbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9" Nov 03 03:29:41.349967 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.349893 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd5d230-0bbe-4410-853d-b149c782dbf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-k95m9\" (UID: \"9bd5d230-0bbe-4410-853d-b149c782dbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9" Nov 03 03:29:41.350601 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.350580 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd5d230-0bbe-4410-853d-b149c782dbf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-k95m9\" (UID: \"9bd5d230-0bbe-4410-853d-b149c782dbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9" Nov 03 03:29:41.352167 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.352148 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd5d230-0bbe-4410-853d-b149c782dbf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-k95m9\" (UID: \"9bd5d230-0bbe-4410-853d-b149c782dbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9" Nov 03 03:29:41.406720 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.406649 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mptl9\" (UniqueName: \"kubernetes.io/projected/9bd5d230-0bbe-4410-853d-b149c782dbf9-kube-api-access-mptl9\") pod \"maas-api-revoke-sub-d3fba056-k95m9\" (UID: \"9bd5d230-0bbe-4410-853d-b149c782dbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9" Nov 03 03:29:41.470870 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.470845 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9" Nov 03 03:29:41.841365 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:29:41.841333 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd5d230_0bbe_4410_853d_b149c782dbf9.slice/crio-7b0c7465df8c049465e70d37f732afa8abe259a1b8fca51cfebbc60cf0693200 WatchSource:0}: Error finding container 7b0c7465df8c049465e70d37f732afa8abe259a1b8fca51cfebbc60cf0693200: Status 404 returned error can't find the container with id 7b0c7465df8c049465e70d37f732afa8abe259a1b8fca51cfebbc60cf0693200 Nov 03 03:29:41.843912 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:41.843888 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9"] Nov 03 03:29:42.127444 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:42.127364 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9" event={"ID":"9bd5d230-0bbe-4410-853d-b149c782dbf9","Type":"ContainerStarted","Data":"7b0c7465df8c049465e70d37f732afa8abe259a1b8fca51cfebbc60cf0693200"} Nov 03 03:29:43.007274 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:29:43.007186 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:29:43.134640 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:43.134599 5463 generic.go:356] "Generic (PLEG): container finished" podID="9bd5d230-0bbe-4410-853d-b149c782dbf9" containerID="f21dde297d208aa292c0a210896be31208dd8fa397165db8d3c0b3b6a812aa43" exitCode=0 Nov 03 03:29:43.134829 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:43.134681 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9" event={"ID":"9bd5d230-0bbe-4410-853d-b149c782dbf9","Type":"ContainerDied","Data":"f21dde297d208aa292c0a210896be31208dd8fa397165db8d3c0b3b6a812aa43"} Nov 03 03:29:44.313021 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:44.312995 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9" Nov 03 03:29:44.382891 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:44.382853 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd5d230-0bbe-4410-853d-b149c782dbf9-kube-api-access-mptl9\" (UniqueName: \"kubernetes.io/projected/9bd5d230-0bbe-4410-853d-b149c782dbf9-kube-api-access-mptl9\") pod \"9bd5d230-0bbe-4410-853d-b149c782dbf9\" (UID: \"9bd5d230-0bbe-4410-853d-b149c782dbf9\") " Nov 03 03:29:44.383057 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:44.382938 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bd5d230-0bbe-4410-853d-b149c782dbf9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd5d230-0bbe-4410-853d-b149c782dbf9-maas-api-openshift-ca\") pod \"9bd5d230-0bbe-4410-853d-b149c782dbf9\" (UID: \"9bd5d230-0bbe-4410-853d-b149c782dbf9\") " Nov 03 03:29:44.383112 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:44.383097 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bd5d230-0bbe-4410-853d-b149c782dbf9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd5d230-0bbe-4410-853d-b149c782dbf9-maas-api-xks-ca\") pod \"9bd5d230-0bbe-4410-853d-b149c782dbf9\" (UID: \"9bd5d230-0bbe-4410-853d-b149c782dbf9\") " Nov 03 03:29:44.383391 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:44.383357 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd5d230-0bbe-4410-853d-b149c782dbf9-maas-api-openshift-ca" pod "9bd5d230-0bbe-4410-853d-b149c782dbf9" (UID: "9bd5d230-0bbe-4410-853d-b149c782dbf9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:29:44.383549 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:44.383518 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd5d230-0bbe-4410-853d-b149c782dbf9-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:29:44.385727 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:44.385694 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd5d230-0bbe-4410-853d-b149c782dbf9-kube-api-access-mptl9" pod "9bd5d230-0bbe-4410-853d-b149c782dbf9" (UID: "9bd5d230-0bbe-4410-853d-b149c782dbf9"). InnerVolumeSpecName "kube-api-access-mptl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:29:44.385833 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:44.385723 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bd5d230-0bbe-4410-853d-b149c782dbf9-maas-api-xks-ca" pod "9bd5d230-0bbe-4410-853d-b149c782dbf9" (UID: "9bd5d230-0bbe-4410-853d-b149c782dbf9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:29:44.485213 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:44.484734 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd5d230-0bbe-4410-853d-b149c782dbf9-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:29:44.485213 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:44.484777 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mptl9\" (UniqueName: \"kubernetes.io/projected/9bd5d230-0bbe-4410-853d-b149c782dbf9-kube-api-access-mptl9\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:29:45.152216 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:45.148264 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9" event={"ID":"9bd5d230-0bbe-4410-853d-b149c782dbf9","Type":"ContainerDied","Data":"7b0c7465df8c049465e70d37f732afa8abe259a1b8fca51cfebbc60cf0693200"} Nov 03 03:29:45.152216 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:45.148329 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b0c7465df8c049465e70d37f732afa8abe259a1b8fca51cfebbc60cf0693200" Nov 03 03:29:45.152216 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:29:45.148419 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9" Nov 03 03:30:00.155523 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:00.155489 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8"] Nov 03 03:30:00.157043 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:00.157012 5463 cpu_manager.go:414] "containerMap: removing container" podUID="9bd5d230-0bbe-4410-853d-b149c782dbf9" containerName="revoke-keys" Nov 03 03:30:00.157231 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:00.157216 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd5d230-0bbe-4410-853d-b149c782dbf9" containerName="revoke-keys" Nov 03 03:30:00.157580 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:00.157564 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd5d230-0bbe-4410-853d-b149c782dbf9" containerName="revoke-keys" Nov 03 03:30:00.166025 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:00.165999 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" Nov 03 03:30:00.210165 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:00.210136 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8"] Nov 03 03:30:00.258125 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:00.258094 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:30:00.258125 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:00.258106 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:30:00.258125 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:00.258106 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:30:00.265512 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:00.265486 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr245\" (UniqueName: \"kubernetes.io/projected/6de8369d-e62d-44ef-bacb-750bac8e012d-kube-api-access-nr245\") pod \"maas-api-key-cleanup-29894610-g6bj8\" (UID: \"6de8369d-e62d-44ef-bacb-750bac8e012d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" Nov 03 03:30:00.265620 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:00.265595 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6de8369d-e62d-44ef-bacb-750bac8e012d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894610-g6bj8\" (UID: \"6de8369d-e62d-44ef-bacb-750bac8e012d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" Nov 03 03:30:00.366744 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:00.366703 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr245\" (UniqueName: \"kubernetes.io/projected/6de8369d-e62d-44ef-bacb-750bac8e012d-kube-api-access-nr245\") pod \"maas-api-key-cleanup-29894610-g6bj8\" (UID: \"6de8369d-e62d-44ef-bacb-750bac8e012d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" Nov 03 03:30:00.366966 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:00.366802 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6de8369d-e62d-44ef-bacb-750bac8e012d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894610-g6bj8\" (UID: \"6de8369d-e62d-44ef-bacb-750bac8e012d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" Nov 03 03:30:00.367665 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:00.367637 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6de8369d-e62d-44ef-bacb-750bac8e012d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894610-g6bj8\" (UID: \"6de8369d-e62d-44ef-bacb-750bac8e012d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" Nov 03 03:30:00.425537 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:00.425448 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr245\" (UniqueName: \"kubernetes.io/projected/6de8369d-e62d-44ef-bacb-750bac8e012d-kube-api-access-nr245\") pod \"maas-api-key-cleanup-29894610-g6bj8\" (UID: \"6de8369d-e62d-44ef-bacb-750bac8e012d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" Nov 03 03:30:00.479937 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:00.479907 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" Nov 03 03:30:00.878551 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:00.878455 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8"] Nov 03 03:30:01.271211 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:01.269228 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" event={"ID":"6de8369d-e62d-44ef-bacb-750bac8e012d","Type":"ContainerStarted","Data":"7190f9bdb11f795dea324e9e83278d484b8f9155cca912cfcfe5659970fc6f17"} Nov 03 03:30:01.271211 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:01.269312 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" event={"ID":"6de8369d-e62d-44ef-bacb-750bac8e012d","Type":"ContainerStarted","Data":"1ac42be226a8f9b9f7ded10055ec8ee42a48f5fd80667fb1a248bdce4e10535a"} Nov 03 03:30:02.329844 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:02.329807 5463 generic.go:356] "Generic (PLEG): container finished" podID="6de8369d-e62d-44ef-bacb-750bac8e012d" containerID="7190f9bdb11f795dea324e9e83278d484b8f9155cca912cfcfe5659970fc6f17" exitCode=60 Nov 03 03:30:02.330511 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:02.330485 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" event={"ID":"6de8369d-e62d-44ef-bacb-750bac8e012d","Type":"ContainerDied","Data":"7190f9bdb11f795dea324e9e83278d484b8f9155cca912cfcfe5659970fc6f17"} Nov 03 03:30:02.330677 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:02.330593 5463 scope.go:122] "RemoveContainer" containerID="7190f9bdb11f795dea324e9e83278d484b8f9155cca912cfcfe5659970fc6f17" Nov 03 03:30:02.387221 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:02.386719 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9bjwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:30:02.387221 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:02.386814 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" podUID="77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:30:03.047073 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:03.046977 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm"] Nov 03 03:30:03.051338 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:03.051312 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eb55f9d-qspfm"] Nov 03 03:30:03.337605 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:03.337523 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" event={"ID":"6de8369d-e62d-44ef-bacb-750bac8e012d","Type":"ContainerStarted","Data":"112b56ac8238ff1dadd4251f4445fe1b6b95e4fca1085616d885672dc32903e5"} Nov 03 03:30:03.453673 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:03.453627 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" podStartSLOduration=3.45361437 podStartE2EDuration="3.45361437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:30:01.40000601 +0000 UTC m=+5883.084115200" watchObservedRunningTime="2026-11-03 03:30:03.45361437 +0000 UTC m=+5885.137723553" Nov 03 03:30:04.345486 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:04.345448 5463 generic.go:356] "Generic (PLEG): container finished" podID="6de8369d-e62d-44ef-bacb-750bac8e012d" containerID="112b56ac8238ff1dadd4251f4445fe1b6b95e4fca1085616d885672dc32903e5" exitCode=60 Nov 03 03:30:04.345910 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:04.345525 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" event={"ID":"6de8369d-e62d-44ef-bacb-750bac8e012d","Type":"ContainerDied","Data":"112b56ac8238ff1dadd4251f4445fe1b6b95e4fca1085616d885672dc32903e5"} Nov 03 03:30:04.345910 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:04.345592 5463 scope.go:122] "RemoveContainer" containerID="7190f9bdb11f795dea324e9e83278d484b8f9155cca912cfcfe5659970fc6f17" Nov 03 03:30:04.345910 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:04.345763 5463 scope.go:122] "RemoveContainer" containerID="112b56ac8238ff1dadd4251f4445fe1b6b95e4fca1085616d885672dc32903e5" Nov 03 03:30:04.346057 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:30:04.346038 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894610-g6bj8_odh-ai-gateway-infra(6de8369d-e62d-44ef-bacb-750bac8e012d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" podUID="6de8369d-e62d-44ef-bacb-750bac8e012d" Nov 03 03:30:04.441954 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:04.441924 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e070a2f-881a-40da-95f4-bc88bd52b096" path="/var/lib/kubelet/pods/2e070a2f-881a-40da-95f4-bc88bd52b096/volumes" Nov 03 03:30:05.353587 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:05.353559 5463 scope.go:122] "RemoveContainer" containerID="112b56ac8238ff1dadd4251f4445fe1b6b95e4fca1085616d885672dc32903e5" Nov 03 03:30:05.354240 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:30:05.353840 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894610-g6bj8_odh-ai-gateway-infra(6de8369d-e62d-44ef-bacb-750bac8e012d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" podUID="6de8369d-e62d-44ef-bacb-750bac8e012d" Nov 03 03:30:13.007047 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:30:13.006984 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:30:16.435666 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:16.435632 5463 scope.go:122] "RemoveContainer" containerID="112b56ac8238ff1dadd4251f4445fe1b6b95e4fca1085616d885672dc32903e5" Nov 03 03:30:17.422708 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:17.422669 5463 generic.go:356] "Generic (PLEG): container finished" podID="6de8369d-e62d-44ef-bacb-750bac8e012d" containerID="0d4705b94326b791d367260f08eee2b2e1a71b7ea31ce708178a20ac6c121444" exitCode=60 Nov 03 03:30:17.422919 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:17.422748 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" event={"ID":"6de8369d-e62d-44ef-bacb-750bac8e012d","Type":"ContainerDied","Data":"0d4705b94326b791d367260f08eee2b2e1a71b7ea31ce708178a20ac6c121444"} Nov 03 03:30:17.422919 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:17.422827 5463 scope.go:122] "RemoveContainer" containerID="112b56ac8238ff1dadd4251f4445fe1b6b95e4fca1085616d885672dc32903e5" Nov 03 03:30:17.423242 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:17.423182 5463 scope.go:122] "RemoveContainer" containerID="0d4705b94326b791d367260f08eee2b2e1a71b7ea31ce708178a20ac6c121444" Nov 03 03:30:17.423535 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:30:17.423513 5463 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894610-g6bj8_odh-ai-gateway-infra(6de8369d-e62d-44ef-bacb-750bac8e012d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" podUID="6de8369d-e62d-44ef-bacb-750bac8e012d" Nov 03 03:30:18.547776 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:18.547737 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8"] Nov 03 03:30:18.698593 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:18.698567 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" Nov 03 03:30:18.798751 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:18.798665 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6de8369d-e62d-44ef-bacb-750bac8e012d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6de8369d-e62d-44ef-bacb-750bac8e012d-maas-api-service-ca\") pod \"6de8369d-e62d-44ef-bacb-750bac8e012d\" (UID: \"6de8369d-e62d-44ef-bacb-750bac8e012d\") " Nov 03 03:30:18.798935 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:18.798768 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de8369d-e62d-44ef-bacb-750bac8e012d-kube-api-access-nr245\" (UniqueName: \"kubernetes.io/projected/6de8369d-e62d-44ef-bacb-750bac8e012d-kube-api-access-nr245\") pod \"6de8369d-e62d-44ef-bacb-750bac8e012d\" (UID: \"6de8369d-e62d-44ef-bacb-750bac8e012d\") " Nov 03 03:30:18.799023 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:18.798989 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de8369d-e62d-44ef-bacb-750bac8e012d-maas-api-service-ca" pod "6de8369d-e62d-44ef-bacb-750bac8e012d" (UID: "6de8369d-e62d-44ef-bacb-750bac8e012d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:30:18.799166 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:18.799148 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6de8369d-e62d-44ef-bacb-750bac8e012d-maas-api-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:30:18.800834 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:18.800815 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de8369d-e62d-44ef-bacb-750bac8e012d-kube-api-access-nr245" pod "6de8369d-e62d-44ef-bacb-750bac8e012d" (UID: "6de8369d-e62d-44ef-bacb-750bac8e012d"). InnerVolumeSpecName "kube-api-access-nr245". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:30:18.899686 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:18.899653 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr245\" (UniqueName: \"kubernetes.io/projected/6de8369d-e62d-44ef-bacb-750bac8e012d-kube-api-access-nr245\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:30:19.436225 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:19.436202 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" Nov 03 03:30:19.436429 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:19.436220 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8" event={"ID":"6de8369d-e62d-44ef-bacb-750bac8e012d","Type":"ContainerDied","Data":"1ac42be226a8f9b9f7ded10055ec8ee42a48f5fd80667fb1a248bdce4e10535a"} Nov 03 03:30:19.436429 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:19.436273 5463 scope.go:122] "RemoveContainer" containerID="0d4705b94326b791d367260f08eee2b2e1a71b7ea31ce708178a20ac6c121444" Nov 03 03:30:19.601117 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:19.601084 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8"] Nov 03 03:30:19.607394 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:19.607367 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894610-g6bj8"] Nov 03 03:30:20.442270 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:20.442236 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6de8369d-e62d-44ef-bacb-750bac8e012d" path="/var/lib/kubelet/pods/6de8369d-e62d-44ef-bacb-750bac8e012d/volumes" Nov 03 03:30:35.274362 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:35.274312 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:30:36.032216 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.024767 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm"] Nov 03 03:30:36.036826 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.033935 5463 cpu_manager.go:414] "containerMap: removing container" podUID="6de8369d-e62d-44ef-bacb-750bac8e012d" containerName="cleanup" Nov 03 03:30:36.036826 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.035269 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de8369d-e62d-44ef-bacb-750bac8e012d" containerName="cleanup" Nov 03 03:30:36.036826 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.035315 5463 cpu_manager.go:414] "containerMap: removing container" podUID="6de8369d-e62d-44ef-bacb-750bac8e012d" containerName="cleanup" Nov 03 03:30:36.036826 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.035324 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de8369d-e62d-44ef-bacb-750bac8e012d" containerName="cleanup" Nov 03 03:30:36.036826 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.035459 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de8369d-e62d-44ef-bacb-750bac8e012d" containerName="cleanup" Nov 03 03:30:36.036826 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.035481 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de8369d-e62d-44ef-bacb-750bac8e012d" containerName="cleanup" Nov 03 03:30:36.043267 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.043241 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm" Nov 03 03:30:36.103215 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.083933 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm"] Nov 03 03:30:36.131814 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.131486 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:30:36.131814 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.131577 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:30:36.131814 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.131713 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:30:36.224695 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.224568 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:30:36.226531 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.226413 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ccee46-86c2-4665-a038-3703352b4979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7w5vm\" (UID: \"65ccee46-86c2-4665-a038-3703352b4979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm" Nov 03 03:30:36.226862 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.226747 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ccee46-86c2-4665-a038-3703352b4979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7w5vm\" (UID: \"65ccee46-86c2-4665-a038-3703352b4979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm" Nov 03 03:30:36.227302 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.227184 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfhn5\" (UniqueName: \"kubernetes.io/projected/65ccee46-86c2-4665-a038-3703352b4979-kube-api-access-tfhn5\") pod \"maas-api-revoke-sub-a4aef7d7-7w5vm\" (UID: \"65ccee46-86c2-4665-a038-3703352b4979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm" Nov 03 03:30:36.328620 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.328532 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfhn5\" (UniqueName: \"kubernetes.io/projected/65ccee46-86c2-4665-a038-3703352b4979-kube-api-access-tfhn5\") pod \"maas-api-revoke-sub-a4aef7d7-7w5vm\" (UID: \"65ccee46-86c2-4665-a038-3703352b4979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm" Nov 03 03:30:36.328620 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.328601 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ccee46-86c2-4665-a038-3703352b4979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7w5vm\" (UID: \"65ccee46-86c2-4665-a038-3703352b4979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm" Nov 03 03:30:36.329039 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.328636 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ccee46-86c2-4665-a038-3703352b4979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7w5vm\" (UID: \"65ccee46-86c2-4665-a038-3703352b4979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm" Nov 03 03:30:36.329280 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.329260 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ccee46-86c2-4665-a038-3703352b4979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7w5vm\" (UID: \"65ccee46-86c2-4665-a038-3703352b4979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm" Nov 03 03:30:36.330978 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.330958 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ccee46-86c2-4665-a038-3703352b4979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7w5vm\" (UID: \"65ccee46-86c2-4665-a038-3703352b4979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm" Nov 03 03:30:36.406213 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.405854 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfhn5\" (UniqueName: \"kubernetes.io/projected/65ccee46-86c2-4665-a038-3703352b4979-kube-api-access-tfhn5\") pod \"maas-api-revoke-sub-a4aef7d7-7w5vm\" (UID: \"65ccee46-86c2-4665-a038-3703352b4979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm" Nov 03 03:30:36.672216 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:36.671351 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm" Nov 03 03:30:37.163006 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:37.162971 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm"] Nov 03 03:30:37.537659 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:37.537622 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm" event={"ID":"65ccee46-86c2-4665-a038-3703352b4979","Type":"ContainerStarted","Data":"1a1021fdabcb01f56d31d947a49c36aab2226b0a516958f008670e1b09e4eba4"} Nov 03 03:30:38.544873 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:38.544778 5463 generic.go:356] "Generic (PLEG): container finished" podID="65ccee46-86c2-4665-a038-3703352b4979" containerID="c4d69b2cb94c906db6cd18973e66ef7532dead56f38fda5a4c81b886909b3b3c" exitCode=0 Nov 03 03:30:38.545270 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:38.544864 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm" event={"ID":"65ccee46-86c2-4665-a038-3703352b4979","Type":"ContainerDied","Data":"c4d69b2cb94c906db6cd18973e66ef7532dead56f38fda5a4c81b886909b3b3c"} Nov 03 03:30:39.684463 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:39.684440 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm" Nov 03 03:30:39.809648 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:39.809581 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65ccee46-86c2-4665-a038-3703352b4979-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ccee46-86c2-4665-a038-3703352b4979-maas-api-xks-ca\") pod \"65ccee46-86c2-4665-a038-3703352b4979\" (UID: \"65ccee46-86c2-4665-a038-3703352b4979\") " Nov 03 03:30:39.809648 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:39.809626 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ccee46-86c2-4665-a038-3703352b4979-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ccee46-86c2-4665-a038-3703352b4979-maas-api-openshift-ca\") pod \"65ccee46-86c2-4665-a038-3703352b4979\" (UID: \"65ccee46-86c2-4665-a038-3703352b4979\") " Nov 03 03:30:39.809882 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:39.809786 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ccee46-86c2-4665-a038-3703352b4979-kube-api-access-tfhn5\" (UniqueName: \"kubernetes.io/projected/65ccee46-86c2-4665-a038-3703352b4979-kube-api-access-tfhn5\") pod \"65ccee46-86c2-4665-a038-3703352b4979\" (UID: \"65ccee46-86c2-4665-a038-3703352b4979\") " Nov 03 03:30:39.810085 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:39.810048 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ccee46-86c2-4665-a038-3703352b4979-maas-api-openshift-ca" pod "65ccee46-86c2-4665-a038-3703352b4979" (UID: "65ccee46-86c2-4665-a038-3703352b4979"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:30:39.811797 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:39.811771 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ccee46-86c2-4665-a038-3703352b4979-maas-api-xks-ca" pod "65ccee46-86c2-4665-a038-3703352b4979" (UID: "65ccee46-86c2-4665-a038-3703352b4979"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:30:39.811914 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:39.811861 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ccee46-86c2-4665-a038-3703352b4979-kube-api-access-tfhn5" pod "65ccee46-86c2-4665-a038-3703352b4979" (UID: "65ccee46-86c2-4665-a038-3703352b4979"). InnerVolumeSpecName "kube-api-access-tfhn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:30:39.910804 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:39.910775 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfhn5\" (UniqueName: \"kubernetes.io/projected/65ccee46-86c2-4665-a038-3703352b4979-kube-api-access-tfhn5\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:30:39.910804 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:39.910801 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ccee46-86c2-4665-a038-3703352b4979-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:30:39.911024 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:39.910813 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ccee46-86c2-4665-a038-3703352b4979-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:30:40.557076 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:40.557048 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm" Nov 03 03:30:40.557256 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:40.557084 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm" event={"ID":"65ccee46-86c2-4665-a038-3703352b4979","Type":"ContainerDied","Data":"1a1021fdabcb01f56d31d947a49c36aab2226b0a516958f008670e1b09e4eba4"} Nov 03 03:30:40.557256 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:40.557125 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a1021fdabcb01f56d31d947a49c36aab2226b0a516958f008670e1b09e4eba4" Nov 03 03:30:43.006870 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:30:43.006825 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:30:57.330961 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.330904 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd"] Nov 03 03:30:57.332213 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.332167 5463 cpu_manager.go:414] "containerMap: removing container" podUID="65ccee46-86c2-4665-a038-3703352b4979" containerName="revoke-keys" Nov 03 03:30:57.332379 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.332362 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ccee46-86c2-4665-a038-3703352b4979" containerName="revoke-keys" Nov 03 03:30:57.332495 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.332483 5463 cpu_manager.go:414] "containerMap: removing container" podUID="6de8369d-e62d-44ef-bacb-750bac8e012d" containerName="cleanup" Nov 03 03:30:57.332595 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.332580 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de8369d-e62d-44ef-bacb-750bac8e012d" containerName="cleanup" Nov 03 03:30:57.332848 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.332833 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ccee46-86c2-4665-a038-3703352b4979" containerName="revoke-keys" Nov 03 03:30:57.332960 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.332948 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de8369d-e62d-44ef-bacb-750bac8e012d" containerName="cleanup" Nov 03 03:30:57.342893 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.342859 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd" Nov 03 03:30:57.410214 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.389162 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd"] Nov 03 03:30:57.415213 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.411710 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rndzd\" (UID: \"8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd" Nov 03 03:30:57.433059 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.433028 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ntbf\" (UniqueName: \"kubernetes.io/projected/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-kube-api-access-9ntbf\") pod \"maas-api-revoke-sub-79805ca9-rndzd\" (UID: \"8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd" Nov 03 03:30:57.434012 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.433988 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rndzd\" (UID: \"8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd" Nov 03 03:30:57.436320 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.436298 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:30:57.436670 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.436613 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:30:57.436819 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.436735 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:30:57.453381 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.453361 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:30:57.535900 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.535866 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rndzd\" (UID: \"8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd" Nov 03 03:30:57.536208 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.536171 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rndzd\" (UID: \"8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd" Nov 03 03:30:57.536400 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.536385 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ntbf\" (UniqueName: \"kubernetes.io/projected/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-kube-api-access-9ntbf\") pod \"maas-api-revoke-sub-79805ca9-rndzd\" (UID: \"8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd" Nov 03 03:30:57.536570 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.536453 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rndzd\" (UID: \"8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd" Nov 03 03:30:57.538670 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.538648 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rndzd\" (UID: \"8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd" Nov 03 03:30:57.623124 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.623033 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ntbf\" (UniqueName: \"kubernetes.io/projected/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-kube-api-access-9ntbf\") pod \"maas-api-revoke-sub-79805ca9-rndzd\" (UID: \"8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd" Nov 03 03:30:57.737145 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:57.737113 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd" Nov 03 03:30:58.224213 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:58.224104 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd"] Nov 03 03:30:58.229422 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:30:58.229390 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2c6f15_9f2c_48c8_9fef_1a8eb82ca04f.slice/crio-dd7a0970040ad7f182c0a1868a6373234f278e77067ef38fa6a51780d0409c50 WatchSource:0}: Error finding container dd7a0970040ad7f182c0a1868a6373234f278e77067ef38fa6a51780d0409c50: Status 404 returned error can't find the container with id dd7a0970040ad7f182c0a1868a6373234f278e77067ef38fa6a51780d0409c50 Nov 03 03:30:58.722965 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:58.722912 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd" event={"ID":"8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f","Type":"ContainerStarted","Data":"0bd4c369ccdcb331531c5b5a7f749ca9c17580fff3a6e5aa29fba5bcb59d5374"} Nov 03 03:30:58.723599 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:58.723577 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd" event={"ID":"8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f","Type":"ContainerStarted","Data":"dd7a0970040ad7f182c0a1868a6373234f278e77067ef38fa6a51780d0409c50"} Nov 03 03:30:58.837664 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:58.837561 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd" podStartSLOduration=1.8375473850000001 podStartE2EDuration="1.837547385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:30:58.837545538 +0000 UTC m=+5940.521654693" watchObservedRunningTime="2026-11-03 03:30:58.837547385 +0000 UTC m=+5940.521656540" Nov 03 03:30:59.730151 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:59.730115 5463 generic.go:356] "Generic (PLEG): container finished" podID="8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f" containerID="0bd4c369ccdcb331531c5b5a7f749ca9c17580fff3a6e5aa29fba5bcb59d5374" exitCode=0 Nov 03 03:30:59.730565 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:30:59.730222 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd" event={"ID":"8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f","Type":"ContainerDied","Data":"0bd4c369ccdcb331531c5b5a7f749ca9c17580fff3a6e5aa29fba5bcb59d5374"} Nov 03 03:31:00.871081 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:00.871056 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd" Nov 03 03:31:00.986683 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:00.986605 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-maas-api-xks-ca\") pod \"8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f\" (UID: \"8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f\") " Nov 03 03:31:00.987028 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:00.987008 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-maas-api-openshift-ca\") pod \"8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f\" (UID: \"8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f\") " Nov 03 03:31:00.987329 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:00.987306 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-kube-api-access-9ntbf\" (UniqueName: \"kubernetes.io/projected/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-kube-api-access-9ntbf\") pod \"8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f\" (UID: \"8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f\") " Nov 03 03:31:00.987972 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:00.987900 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-maas-api-openshift-ca" pod "8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f" (UID: "8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:31:00.989243 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:00.989216 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-maas-api-xks-ca" pod "8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f" (UID: "8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:31:00.989877 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:00.989853 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-kube-api-access-9ntbf" pod "8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f" (UID: "8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f"). InnerVolumeSpecName "kube-api-access-9ntbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:01.088622 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:01.088595 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ntbf\" (UniqueName: \"kubernetes.io/projected/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-kube-api-access-9ntbf\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:31:01.088622 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:01.088621 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:31:01.088757 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:01.088632 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:31:01.741889 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:01.741861 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd" Nov 03 03:31:01.741889 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:01.741876 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd" event={"ID":"8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f","Type":"ContainerDied","Data":"dd7a0970040ad7f182c0a1868a6373234f278e77067ef38fa6a51780d0409c50"} Nov 03 03:31:01.742126 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:01.741918 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd7a0970040ad7f182c0a1868a6373234f278e77067ef38fa6a51780d0409c50" Nov 03 03:31:06.355229 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:06.355179 5463 scope.go:122] "RemoveContainer" containerID="dc84199a6f4d7d2857db1b94379fcdd6730889ee689b9c52713c701494fc0f4e" Nov 03 03:31:06.364093 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:06.364073 5463 scope.go:122] "RemoveContainer" containerID="385469a204907a7e2b8e6ec1b869f959860cd99ff62f9b36438d89fa73f43fe4" Nov 03 03:31:06.374143 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:06.374107 5463 scope.go:122] "RemoveContainer" containerID="25fae792f6076283644db9aedee1eb9966f64e054cd7a2c19e22fdd5cb5f2205" Nov 03 03:31:13.008059 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:31:13.007985 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:31:14.850669 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:14.850626 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv"] Nov 03 03:31:14.851182 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:14.851142 5463 cpu_manager.go:414] "containerMap: removing container" podUID="8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f" containerName="revoke-keys" Nov 03 03:31:14.851182 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:14.851157 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f" containerName="revoke-keys" Nov 03 03:31:14.851373 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:14.851275 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f" containerName="revoke-keys" Nov 03 03:31:14.855957 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:14.855935 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv" Nov 03 03:31:14.926720 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:14.926688 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv"] Nov 03 03:31:14.969215 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:14.966563 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:31:14.969215 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:14.966715 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:31:14.969215 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:14.966795 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:31:14.969215 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:14.967420 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:31:15.027220 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:15.026520 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-lvpzv\" (UID: \"d016ecb8-3e97-4ce3-a419-58ad53a2cf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv" Nov 03 03:31:15.027220 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:15.026589 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-lvpzv\" (UID: \"d016ecb8-3e97-4ce3-a419-58ad53a2cf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv" Nov 03 03:31:15.027220 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:15.026699 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq4zh\" (UniqueName: \"kubernetes.io/projected/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-kube-api-access-rq4zh\") pod \"maas-api-revoke-sub-0b7e8edb-lvpzv\" (UID: \"d016ecb8-3e97-4ce3-a419-58ad53a2cf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv" Nov 03 03:31:15.131270 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:15.131119 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4zh\" (UniqueName: \"kubernetes.io/projected/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-kube-api-access-rq4zh\") pod \"maas-api-revoke-sub-0b7e8edb-lvpzv\" (UID: \"d016ecb8-3e97-4ce3-a419-58ad53a2cf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv" Nov 03 03:31:15.136212 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:15.135081 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-lvpzv\" (UID: \"d016ecb8-3e97-4ce3-a419-58ad53a2cf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv" Nov 03 03:31:15.136212 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:15.135165 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-lvpzv\" (UID: \"d016ecb8-3e97-4ce3-a419-58ad53a2cf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv" Nov 03 03:31:15.136212 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:15.135420 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-lvpzv\" (UID: \"d016ecb8-3e97-4ce3-a419-58ad53a2cf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv" Nov 03 03:31:15.143688 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:15.143663 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-lvpzv\" (UID: \"d016ecb8-3e97-4ce3-a419-58ad53a2cf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv" Nov 03 03:31:15.197906 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:15.197874 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq4zh\" (UniqueName: \"kubernetes.io/projected/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-kube-api-access-rq4zh\") pod \"maas-api-revoke-sub-0b7e8edb-lvpzv\" (UID: \"d016ecb8-3e97-4ce3-a419-58ad53a2cf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv" Nov 03 03:31:15.477214 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:15.476326 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv" Nov 03 03:31:15.964177 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:15.964144 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv"] Nov 03 03:31:16.844519 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:16.844478 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv" event={"ID":"d016ecb8-3e97-4ce3-a419-58ad53a2cf7d","Type":"ContainerStarted","Data":"a156aa6414a59a318909dce3cb0fe2e35df9ac83d6668608c74232b69242bfaf"} Nov 03 03:31:16.844697 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:16.844543 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv" event={"ID":"d016ecb8-3e97-4ce3-a419-58ad53a2cf7d","Type":"ContainerStarted","Data":"ecb8ac193fefbcd0ca245b939bfd583afe53a16579b213ffb5267999d52658be"} Nov 03 03:31:16.961242 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:16.961168 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv" podStartSLOduration=2.961149581 podStartE2EDuration="2.961149581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:31:16.959893447 +0000 UTC m=+5958.644002602" watchObservedRunningTime="2026-11-03 03:31:16.961149581 +0000 UTC m=+5958.645258737" Nov 03 03:31:17.852827 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:17.852787 5463 generic.go:356] "Generic (PLEG): container finished" podID="d016ecb8-3e97-4ce3-a419-58ad53a2cf7d" containerID="a156aa6414a59a318909dce3cb0fe2e35df9ac83d6668608c74232b69242bfaf" exitCode=0 Nov 03 03:31:17.853264 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:17.852862 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv" event={"ID":"d016ecb8-3e97-4ce3-a419-58ad53a2cf7d","Type":"ContainerDied","Data":"a156aa6414a59a318909dce3cb0fe2e35df9ac83d6668608c74232b69242bfaf"} Nov 03 03:31:19.002517 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:19.002488 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv" Nov 03 03:31:19.102239 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:19.102208 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-maas-api-openshift-ca\") pod \"d016ecb8-3e97-4ce3-a419-58ad53a2cf7d\" (UID: \"d016ecb8-3e97-4ce3-a419-58ad53a2cf7d\") " Nov 03 03:31:19.102385 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:19.102255 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-kube-api-access-rq4zh\" (UniqueName: \"kubernetes.io/projected/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-kube-api-access-rq4zh\") pod \"d016ecb8-3e97-4ce3-a419-58ad53a2cf7d\" (UID: \"d016ecb8-3e97-4ce3-a419-58ad53a2cf7d\") " Nov 03 03:31:19.102385 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:19.102366 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-maas-api-xks-ca\") pod \"d016ecb8-3e97-4ce3-a419-58ad53a2cf7d\" (UID: \"d016ecb8-3e97-4ce3-a419-58ad53a2cf7d\") " Nov 03 03:31:19.102551 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:19.102529 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-maas-api-openshift-ca" pod "d016ecb8-3e97-4ce3-a419-58ad53a2cf7d" (UID: "d016ecb8-3e97-4ce3-a419-58ad53a2cf7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:31:19.104409 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:19.104387 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-maas-api-xks-ca" pod "d016ecb8-3e97-4ce3-a419-58ad53a2cf7d" (UID: "d016ecb8-3e97-4ce3-a419-58ad53a2cf7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:31:19.104506 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:19.104408 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-kube-api-access-rq4zh" pod "d016ecb8-3e97-4ce3-a419-58ad53a2cf7d" (UID: "d016ecb8-3e97-4ce3-a419-58ad53a2cf7d"). InnerVolumeSpecName "kube-api-access-rq4zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:19.203578 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:19.203550 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:31:19.203578 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:19.203576 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq4zh\" (UniqueName: \"kubernetes.io/projected/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-kube-api-access-rq4zh\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:31:19.203768 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:19.203586 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:31:19.865891 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:19.865864 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv" Nov 03 03:31:19.866073 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:19.865856 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv" event={"ID":"d016ecb8-3e97-4ce3-a419-58ad53a2cf7d","Type":"ContainerDied","Data":"ecb8ac193fefbcd0ca245b939bfd583afe53a16579b213ffb5267999d52658be"} Nov 03 03:31:19.866073 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:19.865997 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecb8ac193fefbcd0ca245b939bfd583afe53a16579b213ffb5267999d52658be" Nov 03 03:31:26.146821 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.146786 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2"] Nov 03 03:31:26.147270 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.147255 5463 cpu_manager.go:414] "containerMap: removing container" podUID="d016ecb8-3e97-4ce3-a419-58ad53a2cf7d" containerName="revoke-keys" Nov 03 03:31:26.147323 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.147270 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d016ecb8-3e97-4ce3-a419-58ad53a2cf7d" containerName="revoke-keys" Nov 03 03:31:26.147362 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.147345 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="d016ecb8-3e97-4ce3-a419-58ad53a2cf7d" containerName="revoke-keys" Nov 03 03:31:26.150483 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.150468 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2" Nov 03 03:31:26.254129 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.254100 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:31:26.254326 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.254131 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:31:26.254326 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.254183 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:31:26.254326 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.254254 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:31:26.256384 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.256366 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2"] Nov 03 03:31:26.271750 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.271718 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6f1928-5210-40d8-b66c-d77bfba7604e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fmfp2\" (UID: \"9e6f1928-5210-40d8-b66c-d77bfba7604e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2" Nov 03 03:31:26.271750 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.271745 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c99ps\" (UniqueName: \"kubernetes.io/projected/9e6f1928-5210-40d8-b66c-d77bfba7604e-kube-api-access-c99ps\") pod \"maas-api-revoke-sub-3d4156f1-fmfp2\" (UID: \"9e6f1928-5210-40d8-b66c-d77bfba7604e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2" Nov 03 03:31:26.271976 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.271843 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6f1928-5210-40d8-b66c-d77bfba7604e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fmfp2\" (UID: \"9e6f1928-5210-40d8-b66c-d77bfba7604e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2" Nov 03 03:31:26.372463 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.372425 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6f1928-5210-40d8-b66c-d77bfba7604e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fmfp2\" (UID: \"9e6f1928-5210-40d8-b66c-d77bfba7604e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2" Nov 03 03:31:26.372674 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.372467 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c99ps\" (UniqueName: \"kubernetes.io/projected/9e6f1928-5210-40d8-b66c-d77bfba7604e-kube-api-access-c99ps\") pod \"maas-api-revoke-sub-3d4156f1-fmfp2\" (UID: \"9e6f1928-5210-40d8-b66c-d77bfba7604e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2" Nov 03 03:31:26.372674 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.372541 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6f1928-5210-40d8-b66c-d77bfba7604e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fmfp2\" (UID: \"9e6f1928-5210-40d8-b66c-d77bfba7604e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2" Nov 03 03:31:26.373312 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.373284 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6f1928-5210-40d8-b66c-d77bfba7604e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fmfp2\" (UID: \"9e6f1928-5210-40d8-b66c-d77bfba7604e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2" Nov 03 03:31:26.375037 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.375001 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6f1928-5210-40d8-b66c-d77bfba7604e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fmfp2\" (UID: \"9e6f1928-5210-40d8-b66c-d77bfba7604e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2" Nov 03 03:31:26.431280 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.431208 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c99ps\" (UniqueName: \"kubernetes.io/projected/9e6f1928-5210-40d8-b66c-d77bfba7604e-kube-api-access-c99ps\") pod \"maas-api-revoke-sub-3d4156f1-fmfp2\" (UID: \"9e6f1928-5210-40d8-b66c-d77bfba7604e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2" Nov 03 03:31:26.466055 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.466030 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2" Nov 03 03:31:26.843486 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:31:26.843455 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6f1928_5210_40d8_b66c_d77bfba7604e.slice/crio-54e3a9ad0d59cb7754eba821be8d72fe85596114cfa0035d18ea0278974bc82f WatchSource:0}: Error finding container 54e3a9ad0d59cb7754eba821be8d72fe85596114cfa0035d18ea0278974bc82f: Status 404 returned error can't find the container with id 54e3a9ad0d59cb7754eba821be8d72fe85596114cfa0035d18ea0278974bc82f Nov 03 03:31:26.846708 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.846681 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2"] Nov 03 03:31:26.910016 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:26.909980 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2" event={"ID":"9e6f1928-5210-40d8-b66c-d77bfba7604e","Type":"ContainerStarted","Data":"54e3a9ad0d59cb7754eba821be8d72fe85596114cfa0035d18ea0278974bc82f"} Nov 03 03:31:27.917523 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:27.917494 5463 generic.go:356] "Generic (PLEG): container finished" podID="9e6f1928-5210-40d8-b66c-d77bfba7604e" containerID="bf7fdf703396e31561eb49f76bdb4d2eac80c5218f84a5eb6d14ccc443c51bd7" exitCode=0 Nov 03 03:31:27.917905 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:27.917527 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2" event={"ID":"9e6f1928-5210-40d8-b66c-d77bfba7604e","Type":"ContainerDied","Data":"bf7fdf703396e31561eb49f76bdb4d2eac80c5218f84a5eb6d14ccc443c51bd7"} Nov 03 03:31:29.070441 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:29.070410 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2" Nov 03 03:31:29.201819 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:29.201737 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e6f1928-5210-40d8-b66c-d77bfba7604e-kube-api-access-c99ps\" (UniqueName: \"kubernetes.io/projected/9e6f1928-5210-40d8-b66c-d77bfba7604e-kube-api-access-c99ps\") pod \"9e6f1928-5210-40d8-b66c-d77bfba7604e\" (UID: \"9e6f1928-5210-40d8-b66c-d77bfba7604e\") " Nov 03 03:31:29.201981 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:29.201818 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e6f1928-5210-40d8-b66c-d77bfba7604e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6f1928-5210-40d8-b66c-d77bfba7604e-maas-api-openshift-ca\") pod \"9e6f1928-5210-40d8-b66c-d77bfba7604e\" (UID: \"9e6f1928-5210-40d8-b66c-d77bfba7604e\") " Nov 03 03:31:29.202031 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:29.201976 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e6f1928-5210-40d8-b66c-d77bfba7604e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6f1928-5210-40d8-b66c-d77bfba7604e-maas-api-xks-ca\") pod \"9e6f1928-5210-40d8-b66c-d77bfba7604e\" (UID: \"9e6f1928-5210-40d8-b66c-d77bfba7604e\") " Nov 03 03:31:29.202495 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:29.202465 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e6f1928-5210-40d8-b66c-d77bfba7604e-maas-api-openshift-ca" pod "9e6f1928-5210-40d8-b66c-d77bfba7604e" (UID: "9e6f1928-5210-40d8-b66c-d77bfba7604e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:31:29.204306 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:29.204276 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e6f1928-5210-40d8-b66c-d77bfba7604e-maas-api-xks-ca" pod "9e6f1928-5210-40d8-b66c-d77bfba7604e" (UID: "9e6f1928-5210-40d8-b66c-d77bfba7604e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:31:29.204463 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:29.204443 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6f1928-5210-40d8-b66c-d77bfba7604e-kube-api-access-c99ps" pod "9e6f1928-5210-40d8-b66c-d77bfba7604e" (UID: "9e6f1928-5210-40d8-b66c-d77bfba7604e"). InnerVolumeSpecName "kube-api-access-c99ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:29.303117 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:29.303080 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6f1928-5210-40d8-b66c-d77bfba7604e-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:31:29.303117 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:29.303111 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6f1928-5210-40d8-b66c-d77bfba7604e-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:31:29.303310 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:29.303126 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c99ps\" (UniqueName: \"kubernetes.io/projected/9e6f1928-5210-40d8-b66c-d77bfba7604e-kube-api-access-c99ps\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:31:29.929760 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:29.929729 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2" Nov 03 03:31:29.929987 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:29.929764 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2" event={"ID":"9e6f1928-5210-40d8-b66c-d77bfba7604e","Type":"ContainerDied","Data":"54e3a9ad0d59cb7754eba821be8d72fe85596114cfa0035d18ea0278974bc82f"} Nov 03 03:31:29.929987 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:29.929818 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54e3a9ad0d59cb7754eba821be8d72fe85596114cfa0035d18ea0278974bc82f" Nov 03 03:31:35.601527 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:35.601494 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:31:41.005186 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:41.005148 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:31:43.100215 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:31:43.086960 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:31:44.389682 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.389651 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2"] Nov 03 03:31:44.441439 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.441412 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmfp2"] Nov 03 03:31:44.517238 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.514838 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4"] Nov 03 03:31:44.517238 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.515883 5463 cpu_manager.go:414] "containerMap: removing container" podUID="9e6f1928-5210-40d8-b66c-d77bfba7604e" containerName="revoke-keys" Nov 03 03:31:44.517238 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.515906 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e6f1928-5210-40d8-b66c-d77bfba7604e" containerName="revoke-keys" Nov 03 03:31:44.517238 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.516103 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e6f1928-5210-40d8-b66c-d77bfba7604e" containerName="revoke-keys" Nov 03 03:31:44.522611 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.522581 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4" Nov 03 03:31:44.561313 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.561285 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ec08d0-fc96-431b-9408-0c6357a710b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vbzr4\" (UID: \"c4ec08d0-fc96-431b-9408-0c6357a710b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4" Nov 03 03:31:44.561452 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.561335 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ec08d0-fc96-431b-9408-0c6357a710b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vbzr4\" (UID: \"c4ec08d0-fc96-431b-9408-0c6357a710b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4" Nov 03 03:31:44.561528 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.561463 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzqnw\" (UniqueName: \"kubernetes.io/projected/c4ec08d0-fc96-431b-9408-0c6357a710b9-kube-api-access-qzqnw\") pod \"maas-api-revoke-sub-3d4156f1-vbzr4\" (UID: \"c4ec08d0-fc96-431b-9408-0c6357a710b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4" Nov 03 03:31:44.570602 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.570574 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4"] Nov 03 03:31:44.575418 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.575396 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:31:44.575551 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.575462 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:31:44.575551 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.575471 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:31:44.575677 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.575614 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:31:44.662292 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.662266 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzqnw\" (UniqueName: \"kubernetes.io/projected/c4ec08d0-fc96-431b-9408-0c6357a710b9-kube-api-access-qzqnw\") pod \"maas-api-revoke-sub-3d4156f1-vbzr4\" (UID: \"c4ec08d0-fc96-431b-9408-0c6357a710b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4" Nov 03 03:31:44.662421 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.662322 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ec08d0-fc96-431b-9408-0c6357a710b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vbzr4\" (UID: \"c4ec08d0-fc96-431b-9408-0c6357a710b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4" Nov 03 03:31:44.662421 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.662355 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ec08d0-fc96-431b-9408-0c6357a710b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vbzr4\" (UID: \"c4ec08d0-fc96-431b-9408-0c6357a710b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4" Nov 03 03:31:44.662950 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.662929 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ec08d0-fc96-431b-9408-0c6357a710b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vbzr4\" (UID: \"c4ec08d0-fc96-431b-9408-0c6357a710b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4" Nov 03 03:31:44.664722 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.664702 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ec08d0-fc96-431b-9408-0c6357a710b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vbzr4\" (UID: \"c4ec08d0-fc96-431b-9408-0c6357a710b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4" Nov 03 03:31:44.719304 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.719273 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzqnw\" (UniqueName: \"kubernetes.io/projected/c4ec08d0-fc96-431b-9408-0c6357a710b9-kube-api-access-qzqnw\") pod \"maas-api-revoke-sub-3d4156f1-vbzr4\" (UID: \"c4ec08d0-fc96-431b-9408-0c6357a710b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4" Nov 03 03:31:44.836090 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:44.836060 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4" Nov 03 03:31:45.205168 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:31:45.205133 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ec08d0_fc96_431b_9408_0c6357a710b9.slice/crio-5c97120d8b8f142bca5ac813361c33a40d6f9f3f2aefc3989f493a53c6d958b5 WatchSource:0}: Error finding container 5c97120d8b8f142bca5ac813361c33a40d6f9f3f2aefc3989f493a53c6d958b5: Status 404 returned error can't find the container with id 5c97120d8b8f142bca5ac813361c33a40d6f9f3f2aefc3989f493a53c6d958b5 Nov 03 03:31:45.207164 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:45.207142 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4"] Nov 03 03:31:46.058957 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:46.058928 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4" event={"ID":"c4ec08d0-fc96-431b-9408-0c6357a710b9","Type":"ContainerStarted","Data":"d52b6b4a9997185a51ef3de3f53dc4b6779ea545e81d3427e5660b10bf1c8fc6"} Nov 03 03:31:46.059291 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:46.058985 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4" event={"ID":"c4ec08d0-fc96-431b-9408-0c6357a710b9","Type":"ContainerStarted","Data":"5c97120d8b8f142bca5ac813361c33a40d6f9f3f2aefc3989f493a53c6d958b5"} Nov 03 03:31:46.442236 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:46.442186 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e6f1928-5210-40d8-b66c-d77bfba7604e" path="/var/lib/kubelet/pods/9e6f1928-5210-40d8-b66c-d77bfba7604e/volumes" Nov 03 03:31:47.065790 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:47.065758 5463 generic.go:356] "Generic (PLEG): container finished" podID="c4ec08d0-fc96-431b-9408-0c6357a710b9" containerID="d52b6b4a9997185a51ef3de3f53dc4b6779ea545e81d3427e5660b10bf1c8fc6" exitCode=0 Nov 03 03:31:47.066248 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:47.065829 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4" event={"ID":"c4ec08d0-fc96-431b-9408-0c6357a710b9","Type":"ContainerDied","Data":"d52b6b4a9997185a51ef3de3f53dc4b6779ea545e81d3427e5660b10bf1c8fc6"} Nov 03 03:31:47.210712 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:47.210686 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4" Nov 03 03:31:47.284593 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:47.284567 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4ec08d0-fc96-431b-9408-0c6357a710b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ec08d0-fc96-431b-9408-0c6357a710b9-maas-api-openshift-ca\") pod \"c4ec08d0-fc96-431b-9408-0c6357a710b9\" (UID: \"c4ec08d0-fc96-431b-9408-0c6357a710b9\") " Nov 03 03:31:47.284746 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:47.284634 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4ec08d0-fc96-431b-9408-0c6357a710b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ec08d0-fc96-431b-9408-0c6357a710b9-maas-api-xks-ca\") pod \"c4ec08d0-fc96-431b-9408-0c6357a710b9\" (UID: \"c4ec08d0-fc96-431b-9408-0c6357a710b9\") " Nov 03 03:31:47.284816 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:47.284775 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ec08d0-fc96-431b-9408-0c6357a710b9-kube-api-access-qzqnw\" (UniqueName: \"kubernetes.io/projected/c4ec08d0-fc96-431b-9408-0c6357a710b9-kube-api-access-qzqnw\") pod \"c4ec08d0-fc96-431b-9408-0c6357a710b9\" (UID: \"c4ec08d0-fc96-431b-9408-0c6357a710b9\") " Nov 03 03:31:47.284990 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:47.284962 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ec08d0-fc96-431b-9408-0c6357a710b9-maas-api-openshift-ca" pod "c4ec08d0-fc96-431b-9408-0c6357a710b9" (UID: "c4ec08d0-fc96-431b-9408-0c6357a710b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:31:47.285059 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:47.285046 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ec08d0-fc96-431b-9408-0c6357a710b9-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:31:47.286609 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:47.286587 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4ec08d0-fc96-431b-9408-0c6357a710b9-maas-api-xks-ca" pod "c4ec08d0-fc96-431b-9408-0c6357a710b9" (UID: "c4ec08d0-fc96-431b-9408-0c6357a710b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:31:47.286794 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:47.286775 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ec08d0-fc96-431b-9408-0c6357a710b9-kube-api-access-qzqnw" pod "c4ec08d0-fc96-431b-9408-0c6357a710b9" (UID: "c4ec08d0-fc96-431b-9408-0c6357a710b9"). InnerVolumeSpecName "kube-api-access-qzqnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:47.386497 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:47.386431 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ec08d0-fc96-431b-9408-0c6357a710b9-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:31:47.386497 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:47.386457 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzqnw\" (UniqueName: \"kubernetes.io/projected/c4ec08d0-fc96-431b-9408-0c6357a710b9-kube-api-access-qzqnw\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:31:48.072377 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:48.072346 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4" Nov 03 03:31:48.072857 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:48.072396 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4" event={"ID":"c4ec08d0-fc96-431b-9408-0c6357a710b9","Type":"ContainerDied","Data":"5c97120d8b8f142bca5ac813361c33a40d6f9f3f2aefc3989f493a53c6d958b5"} Nov 03 03:31:48.072857 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:31:48.072454 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c97120d8b8f142bca5ac813361c33a40d6f9f3f2aefc3989f493a53c6d958b5" Nov 03 03:32:03.490133 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:03.490098 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 03:32:03.497032 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:03.497009 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 03:32:13.011361 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:32:13.011306 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:32:19.487557 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:19.487522 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k"] Nov 03 03:32:19.488030 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:19.488000 5463 cpu_manager.go:414] "containerMap: removing container" podUID="c4ec08d0-fc96-431b-9408-0c6357a710b9" containerName="revoke-keys" Nov 03 03:32:19.488030 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:19.488012 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ec08d0-fc96-431b-9408-0c6357a710b9" containerName="revoke-keys" Nov 03 03:32:19.488119 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:19.488108 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ec08d0-fc96-431b-9408-0c6357a710b9" containerName="revoke-keys" Nov 03 03:32:19.491141 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:19.491125 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k" Nov 03 03:32:19.543468 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:19.543391 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k"] Nov 03 03:32:19.585971 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:19.585938 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c325533f-3347-4531-b023-1fdd3bf2eafc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-fph7k\" (UID: \"c325533f-3347-4531-b023-1fdd3bf2eafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k" Nov 03 03:32:19.586118 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:19.585981 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zptq7\" (UniqueName: \"kubernetes.io/projected/c325533f-3347-4531-b023-1fdd3bf2eafc-kube-api-access-zptq7\") pod \"maas-api-revoke-sub-e2fc9c5b-fph7k\" (UID: \"c325533f-3347-4531-b023-1fdd3bf2eafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k" Nov 03 03:32:19.586166 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:19.586116 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c325533f-3347-4531-b023-1fdd3bf2eafc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-fph7k\" (UID: \"c325533f-3347-4531-b023-1fdd3bf2eafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k" Nov 03 03:32:19.590978 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:19.590948 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:32:19.590978 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:19.590960 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:32:19.591161 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:19.591012 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:32:19.594040 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:19.594024 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:32:19.687090 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:19.687058 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zptq7\" (UniqueName: \"kubernetes.io/projected/c325533f-3347-4531-b023-1fdd3bf2eafc-kube-api-access-zptq7\") pod \"maas-api-revoke-sub-e2fc9c5b-fph7k\" (UID: \"c325533f-3347-4531-b023-1fdd3bf2eafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k" Nov 03 03:32:19.687307 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:19.687128 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c325533f-3347-4531-b023-1fdd3bf2eafc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-fph7k\" (UID: \"c325533f-3347-4531-b023-1fdd3bf2eafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k" Nov 03 03:32:19.687307 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:19.687213 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c325533f-3347-4531-b023-1fdd3bf2eafc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-fph7k\" (UID: \"c325533f-3347-4531-b023-1fdd3bf2eafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k" Nov 03 03:32:19.687809 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:19.687790 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c325533f-3347-4531-b023-1fdd3bf2eafc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-fph7k\" (UID: \"c325533f-3347-4531-b023-1fdd3bf2eafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k" Nov 03 03:32:19.689610 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:19.689588 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c325533f-3347-4531-b023-1fdd3bf2eafc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-fph7k\" (UID: \"c325533f-3347-4531-b023-1fdd3bf2eafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k" Nov 03 03:32:19.744245 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:19.744146 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zptq7\" (UniqueName: \"kubernetes.io/projected/c325533f-3347-4531-b023-1fdd3bf2eafc-kube-api-access-zptq7\") pod \"maas-api-revoke-sub-e2fc9c5b-fph7k\" (UID: \"c325533f-3347-4531-b023-1fdd3bf2eafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k" Nov 03 03:32:19.806343 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:19.806310 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k" Nov 03 03:32:20.172935 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:32:20.172901 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc325533f_3347_4531_b023_1fdd3bf2eafc.slice/crio-e3f1d65ad8ac50a1da4c0f9fa198503e3549c96b1de90a26b65bd441aed6c11d WatchSource:0}: Error finding container e3f1d65ad8ac50a1da4c0f9fa198503e3549c96b1de90a26b65bd441aed6c11d: Status 404 returned error can't find the container with id e3f1d65ad8ac50a1da4c0f9fa198503e3549c96b1de90a26b65bd441aed6c11d Nov 03 03:32:20.174793 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:20.174766 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k"] Nov 03 03:32:20.264201 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:20.264168 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k" event={"ID":"c325533f-3347-4531-b023-1fdd3bf2eafc","Type":"ContainerStarted","Data":"e3f1d65ad8ac50a1da4c0f9fa198503e3549c96b1de90a26b65bd441aed6c11d"} Nov 03 03:32:21.270970 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:21.270937 5463 generic.go:356] "Generic (PLEG): container finished" podID="c325533f-3347-4531-b023-1fdd3bf2eafc" containerID="41b2f7fb22c16e46a8610490745c3ffb7f76f1e56b54f766c7f46dc67507f33e" exitCode=0 Nov 03 03:32:21.271391 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:21.271030 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k" event={"ID":"c325533f-3347-4531-b023-1fdd3bf2eafc","Type":"ContainerDied","Data":"41b2f7fb22c16e46a8610490745c3ffb7f76f1e56b54f766c7f46dc67507f33e"} Nov 03 03:32:22.408675 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:22.408654 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k" Nov 03 03:32:22.514929 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:22.514896 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c325533f-3347-4531-b023-1fdd3bf2eafc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c325533f-3347-4531-b023-1fdd3bf2eafc-maas-api-xks-ca\") pod \"c325533f-3347-4531-b023-1fdd3bf2eafc\" (UID: \"c325533f-3347-4531-b023-1fdd3bf2eafc\") " Nov 03 03:32:22.515104 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:22.514946 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c325533f-3347-4531-b023-1fdd3bf2eafc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c325533f-3347-4531-b023-1fdd3bf2eafc-maas-api-openshift-ca\") pod \"c325533f-3347-4531-b023-1fdd3bf2eafc\" (UID: \"c325533f-3347-4531-b023-1fdd3bf2eafc\") " Nov 03 03:32:22.515104 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:22.515024 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c325533f-3347-4531-b023-1fdd3bf2eafc-kube-api-access-zptq7\" (UniqueName: \"kubernetes.io/projected/c325533f-3347-4531-b023-1fdd3bf2eafc-kube-api-access-zptq7\") pod \"c325533f-3347-4531-b023-1fdd3bf2eafc\" (UID: \"c325533f-3347-4531-b023-1fdd3bf2eafc\") " Nov 03 03:32:22.515385 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:22.515361 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c325533f-3347-4531-b023-1fdd3bf2eafc-maas-api-openshift-ca" pod "c325533f-3347-4531-b023-1fdd3bf2eafc" (UID: "c325533f-3347-4531-b023-1fdd3bf2eafc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:32:22.517004 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:22.516983 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c325533f-3347-4531-b023-1fdd3bf2eafc-maas-api-xks-ca" pod "c325533f-3347-4531-b023-1fdd3bf2eafc" (UID: "c325533f-3347-4531-b023-1fdd3bf2eafc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:32:22.517095 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:22.517031 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c325533f-3347-4531-b023-1fdd3bf2eafc-kube-api-access-zptq7" pod "c325533f-3347-4531-b023-1fdd3bf2eafc" (UID: "c325533f-3347-4531-b023-1fdd3bf2eafc"). InnerVolumeSpecName "kube-api-access-zptq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:32:22.616693 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:22.616611 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zptq7\" (UniqueName: \"kubernetes.io/projected/c325533f-3347-4531-b023-1fdd3bf2eafc-kube-api-access-zptq7\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:32:22.616693 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:22.616642 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c325533f-3347-4531-b023-1fdd3bf2eafc-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:32:22.616693 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:22.616653 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c325533f-3347-4531-b023-1fdd3bf2eafc-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:32:23.284466 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:23.284435 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k" Nov 03 03:32:23.284662 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:23.284435 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k" event={"ID":"c325533f-3347-4531-b023-1fdd3bf2eafc","Type":"ContainerDied","Data":"e3f1d65ad8ac50a1da4c0f9fa198503e3549c96b1de90a26b65bd441aed6c11d"} Nov 03 03:32:23.284662 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:23.284574 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3f1d65ad8ac50a1da4c0f9fa198503e3549c96b1de90a26b65bd441aed6c11d" Nov 03 03:32:30.253885 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.253846 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz"] Nov 03 03:32:30.254472 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.254392 5463 cpu_manager.go:414] "containerMap: removing container" podUID="c325533f-3347-4531-b023-1fdd3bf2eafc" containerName="revoke-keys" Nov 03 03:32:30.254472 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.254407 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c325533f-3347-4531-b023-1fdd3bf2eafc" containerName="revoke-keys" Nov 03 03:32:30.254608 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.254491 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="c325533f-3347-4531-b023-1fdd3bf2eafc" containerName="revoke-keys" Nov 03 03:32:30.261716 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.261694 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz" Nov 03 03:32:30.309370 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.309341 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz"] Nov 03 03:32:30.356894 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.356867 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:32:30.356894 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.356866 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:32:30.357097 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.356925 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:32:30.357097 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.357005 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:32:30.388253 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.388227 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04cf8753-4e9d-443d-86ba-9f9d49b7affd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-56mgz\" (UID: \"04cf8753-4e9d-443d-86ba-9f9d49b7affd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz" Nov 03 03:32:30.388390 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.388284 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsvpc\" (UniqueName: \"kubernetes.io/projected/04cf8753-4e9d-443d-86ba-9f9d49b7affd-kube-api-access-wsvpc\") pod \"maas-api-revoke-sub-5a29efb3-56mgz\" (UID: \"04cf8753-4e9d-443d-86ba-9f9d49b7affd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz" Nov 03 03:32:30.388441 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.388426 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04cf8753-4e9d-443d-86ba-9f9d49b7affd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-56mgz\" (UID: \"04cf8753-4e9d-443d-86ba-9f9d49b7affd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz" Nov 03 03:32:30.489130 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.489102 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsvpc\" (UniqueName: \"kubernetes.io/projected/04cf8753-4e9d-443d-86ba-9f9d49b7affd-kube-api-access-wsvpc\") pod \"maas-api-revoke-sub-5a29efb3-56mgz\" (UID: \"04cf8753-4e9d-443d-86ba-9f9d49b7affd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz" Nov 03 03:32:30.489317 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.489155 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04cf8753-4e9d-443d-86ba-9f9d49b7affd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-56mgz\" (UID: \"04cf8753-4e9d-443d-86ba-9f9d49b7affd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz" Nov 03 03:32:30.489317 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.489226 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04cf8753-4e9d-443d-86ba-9f9d49b7affd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-56mgz\" (UID: \"04cf8753-4e9d-443d-86ba-9f9d49b7affd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz" Nov 03 03:32:30.489835 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.489802 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04cf8753-4e9d-443d-86ba-9f9d49b7affd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-56mgz\" (UID: \"04cf8753-4e9d-443d-86ba-9f9d49b7affd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz" Nov 03 03:32:30.491593 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.491568 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04cf8753-4e9d-443d-86ba-9f9d49b7affd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-56mgz\" (UID: \"04cf8753-4e9d-443d-86ba-9f9d49b7affd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz" Nov 03 03:32:30.546262 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.546213 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsvpc\" (UniqueName: \"kubernetes.io/projected/04cf8753-4e9d-443d-86ba-9f9d49b7affd-kube-api-access-wsvpc\") pod \"maas-api-revoke-sub-5a29efb3-56mgz\" (UID: \"04cf8753-4e9d-443d-86ba-9f9d49b7affd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz" Nov 03 03:32:30.579312 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.579290 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz" Nov 03 03:32:30.952078 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:32:30.952046 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cf8753_4e9d_443d_86ba_9f9d49b7affd.slice/crio-bd00c14f03d6cfcb622bee16c8823513de451e4d7d245c80f1fc785fcafe5b8b WatchSource:0}: Error finding container bd00c14f03d6cfcb622bee16c8823513de451e4d7d245c80f1fc785fcafe5b8b: Status 404 returned error can't find the container with id bd00c14f03d6cfcb622bee16c8823513de451e4d7d245c80f1fc785fcafe5b8b Nov 03 03:32:30.954548 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:30.954520 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz"] Nov 03 03:32:31.329983 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:31.329956 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz" event={"ID":"04cf8753-4e9d-443d-86ba-9f9d49b7affd","Type":"ContainerStarted","Data":"22b3665b1dbaf79779553975c7c248571ef8f064d56a1309f815c8933c64b757"} Nov 03 03:32:31.330346 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:31.330008 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz" event={"ID":"04cf8753-4e9d-443d-86ba-9f9d49b7affd","Type":"ContainerStarted","Data":"bd00c14f03d6cfcb622bee16c8823513de451e4d7d245c80f1fc785fcafe5b8b"} Nov 03 03:32:31.444815 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:31.444760 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz" podStartSLOduration=1.444747142 podStartE2EDuration="1.444747142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:32:31.443924335 +0000 UTC m=+6033.128033491" watchObservedRunningTime="2026-11-03 03:32:31.444747142 +0000 UTC m=+6033.128856297" Nov 03 03:32:32.336874 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:32.336839 5463 generic.go:356] "Generic (PLEG): container finished" podID="04cf8753-4e9d-443d-86ba-9f9d49b7affd" containerID="22b3665b1dbaf79779553975c7c248571ef8f064d56a1309f815c8933c64b757" exitCode=0 Nov 03 03:32:32.337308 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:32.336914 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz" event={"ID":"04cf8753-4e9d-443d-86ba-9f9d49b7affd","Type":"ContainerDied","Data":"22b3665b1dbaf79779553975c7c248571ef8f064d56a1309f815c8933c64b757"} Nov 03 03:32:33.506234 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:33.506171 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz" Nov 03 03:32:33.620269 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:33.619531 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04cf8753-4e9d-443d-86ba-9f9d49b7affd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04cf8753-4e9d-443d-86ba-9f9d49b7affd-maas-api-xks-ca\") pod \"04cf8753-4e9d-443d-86ba-9f9d49b7affd\" (UID: \"04cf8753-4e9d-443d-86ba-9f9d49b7affd\") " Nov 03 03:32:33.620269 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:33.619625 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04cf8753-4e9d-443d-86ba-9f9d49b7affd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04cf8753-4e9d-443d-86ba-9f9d49b7affd-maas-api-openshift-ca\") pod \"04cf8753-4e9d-443d-86ba-9f9d49b7affd\" (UID: \"04cf8753-4e9d-443d-86ba-9f9d49b7affd\") " Nov 03 03:32:33.620269 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:33.619743 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04cf8753-4e9d-443d-86ba-9f9d49b7affd-kube-api-access-wsvpc\" (UniqueName: \"kubernetes.io/projected/04cf8753-4e9d-443d-86ba-9f9d49b7affd-kube-api-access-wsvpc\") pod \"04cf8753-4e9d-443d-86ba-9f9d49b7affd\" (UID: \"04cf8753-4e9d-443d-86ba-9f9d49b7affd\") " Nov 03 03:32:33.620889 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:33.620837 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04cf8753-4e9d-443d-86ba-9f9d49b7affd-maas-api-openshift-ca" pod "04cf8753-4e9d-443d-86ba-9f9d49b7affd" (UID: "04cf8753-4e9d-443d-86ba-9f9d49b7affd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:32:33.631760 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:33.631715 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04cf8753-4e9d-443d-86ba-9f9d49b7affd-kube-api-access-wsvpc" pod "04cf8753-4e9d-443d-86ba-9f9d49b7affd" (UID: "04cf8753-4e9d-443d-86ba-9f9d49b7affd"). InnerVolumeSpecName "kube-api-access-wsvpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:32:33.635317 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:33.635280 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04cf8753-4e9d-443d-86ba-9f9d49b7affd-maas-api-xks-ca" pod "04cf8753-4e9d-443d-86ba-9f9d49b7affd" (UID: "04cf8753-4e9d-443d-86ba-9f9d49b7affd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:32:33.724985 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:33.724887 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04cf8753-4e9d-443d-86ba-9f9d49b7affd-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:32:33.724985 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:33.724922 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04cf8753-4e9d-443d-86ba-9f9d49b7affd-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:32:33.724985 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:33.724939 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsvpc\" (UniqueName: \"kubernetes.io/projected/04cf8753-4e9d-443d-86ba-9f9d49b7affd-kube-api-access-wsvpc\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:32:34.374553 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:34.374513 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz" event={"ID":"04cf8753-4e9d-443d-86ba-9f9d49b7affd","Type":"ContainerDied","Data":"bd00c14f03d6cfcb622bee16c8823513de451e4d7d245c80f1fc785fcafe5b8b"} Nov 03 03:32:34.374845 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:34.374823 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd00c14f03d6cfcb622bee16c8823513de451e4d7d245c80f1fc785fcafe5b8b" Nov 03 03:32:34.375044 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:34.375027 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz" Nov 03 03:32:35.994094 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:35.994059 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:32:43.006400 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:32:43.006317 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:32:47.716590 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:47.716552 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4"] Nov 03 03:32:47.769662 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:47.769619 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vbzr4"] Nov 03 03:32:47.840479 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:47.840445 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b"] Nov 03 03:32:47.840914 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:47.840897 5463 cpu_manager.go:414] "containerMap: removing container" podUID="04cf8753-4e9d-443d-86ba-9f9d49b7affd" containerName="revoke-keys" Nov 03 03:32:47.840914 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:47.840912 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04cf8753-4e9d-443d-86ba-9f9d49b7affd" containerName="revoke-keys" Nov 03 03:32:47.841053 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:47.840997 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="04cf8753-4e9d-443d-86ba-9f9d49b7affd" containerName="revoke-keys" Nov 03 03:32:47.844145 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:47.844121 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b" Nov 03 03:32:47.906864 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:47.902344 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:32:47.906864 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:47.902500 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:32:47.906864 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:47.904406 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b"] Nov 03 03:32:47.938019 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:47.937988 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:32:47.938019 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:47.937997 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:32:47.960253 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:47.960220 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66cb95f7-78f5-425d-ab77-8507d378803c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5t58b\" (UID: \"66cb95f7-78f5-425d-ab77-8507d378803c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b" Nov 03 03:32:47.960435 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:47.960333 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktq2z\" (UniqueName: \"kubernetes.io/projected/66cb95f7-78f5-425d-ab77-8507d378803c-kube-api-access-ktq2z\") pod \"maas-api-revoke-sub-3d4156f1-5t58b\" (UID: \"66cb95f7-78f5-425d-ab77-8507d378803c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b" Nov 03 03:32:47.960509 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:47.960490 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66cb95f7-78f5-425d-ab77-8507d378803c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5t58b\" (UID: \"66cb95f7-78f5-425d-ab77-8507d378803c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b" Nov 03 03:32:48.061980 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:48.061882 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66cb95f7-78f5-425d-ab77-8507d378803c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5t58b\" (UID: \"66cb95f7-78f5-425d-ab77-8507d378803c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b" Nov 03 03:32:48.061980 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:48.061972 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktq2z\" (UniqueName: \"kubernetes.io/projected/66cb95f7-78f5-425d-ab77-8507d378803c-kube-api-access-ktq2z\") pod \"maas-api-revoke-sub-3d4156f1-5t58b\" (UID: \"66cb95f7-78f5-425d-ab77-8507d378803c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b" Nov 03 03:32:48.062265 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:48.062120 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66cb95f7-78f5-425d-ab77-8507d378803c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5t58b\" (UID: \"66cb95f7-78f5-425d-ab77-8507d378803c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b" Nov 03 03:32:48.062899 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:48.062868 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66cb95f7-78f5-425d-ab77-8507d378803c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5t58b\" (UID: \"66cb95f7-78f5-425d-ab77-8507d378803c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b" Nov 03 03:32:48.064532 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:48.064503 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66cb95f7-78f5-425d-ab77-8507d378803c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5t58b\" (UID: \"66cb95f7-78f5-425d-ab77-8507d378803c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b" Nov 03 03:32:48.120033 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:48.119993 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktq2z\" (UniqueName: \"kubernetes.io/projected/66cb95f7-78f5-425d-ab77-8507d378803c-kube-api-access-ktq2z\") pod \"maas-api-revoke-sub-3d4156f1-5t58b\" (UID: \"66cb95f7-78f5-425d-ab77-8507d378803c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b" Nov 03 03:32:48.186377 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:48.186343 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b" Nov 03 03:32:48.441873 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:48.441838 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4ec08d0-fc96-431b-9408-0c6357a710b9" path="/var/lib/kubelet/pods/c4ec08d0-fc96-431b-9408-0c6357a710b9/volumes" Nov 03 03:32:48.556762 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:32:48.556731 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66cb95f7_78f5_425d_ab77_8507d378803c.slice/crio-73c76bc838b3110eb8439cf0784cac16d786ef5943014ded7ad3fe375a4c7019 WatchSource:0}: Error finding container 73c76bc838b3110eb8439cf0784cac16d786ef5943014ded7ad3fe375a4c7019: Status 404 returned error can't find the container with id 73c76bc838b3110eb8439cf0784cac16d786ef5943014ded7ad3fe375a4c7019 Nov 03 03:32:48.562612 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:48.559044 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b"] Nov 03 03:32:49.463645 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:49.463617 5463 generic.go:356] "Generic (PLEG): container finished" podID="66cb95f7-78f5-425d-ab77-8507d378803c" containerID="1e9f4708c68d5b45fa47dece58dfb27bab45c39693fc5519a93fce12b689e0c1" exitCode=0 Nov 03 03:32:49.464041 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:49.463704 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b" event={"ID":"66cb95f7-78f5-425d-ab77-8507d378803c","Type":"ContainerDied","Data":"1e9f4708c68d5b45fa47dece58dfb27bab45c39693fc5519a93fce12b689e0c1"} Nov 03 03:32:49.464041 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:49.463811 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b" event={"ID":"66cb95f7-78f5-425d-ab77-8507d378803c","Type":"ContainerStarted","Data":"73c76bc838b3110eb8439cf0784cac16d786ef5943014ded7ad3fe375a4c7019"} Nov 03 03:32:50.620478 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:50.620452 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b" Nov 03 03:32:50.687918 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:50.687894 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66cb95f7-78f5-425d-ab77-8507d378803c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66cb95f7-78f5-425d-ab77-8507d378803c-maas-api-openshift-ca\") pod \"66cb95f7-78f5-425d-ab77-8507d378803c\" (UID: \"66cb95f7-78f5-425d-ab77-8507d378803c\") " Nov 03 03:32:50.688085 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:50.687983 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66cb95f7-78f5-425d-ab77-8507d378803c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66cb95f7-78f5-425d-ab77-8507d378803c-maas-api-xks-ca\") pod \"66cb95f7-78f5-425d-ab77-8507d378803c\" (UID: \"66cb95f7-78f5-425d-ab77-8507d378803c\") " Nov 03 03:32:50.688085 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:50.688049 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66cb95f7-78f5-425d-ab77-8507d378803c-kube-api-access-ktq2z\" (UniqueName: \"kubernetes.io/projected/66cb95f7-78f5-425d-ab77-8507d378803c-kube-api-access-ktq2z\") pod \"66cb95f7-78f5-425d-ab77-8507d378803c\" (UID: \"66cb95f7-78f5-425d-ab77-8507d378803c\") " Nov 03 03:32:50.688309 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:50.688286 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66cb95f7-78f5-425d-ab77-8507d378803c-maas-api-openshift-ca" pod "66cb95f7-78f5-425d-ab77-8507d378803c" (UID: "66cb95f7-78f5-425d-ab77-8507d378803c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:32:50.688396 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:50.688382 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66cb95f7-78f5-425d-ab77-8507d378803c-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:32:50.690180 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:50.690157 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66cb95f7-78f5-425d-ab77-8507d378803c-kube-api-access-ktq2z" pod "66cb95f7-78f5-425d-ab77-8507d378803c" (UID: "66cb95f7-78f5-425d-ab77-8507d378803c"). InnerVolumeSpecName "kube-api-access-ktq2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:32:50.690320 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:50.690201 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66cb95f7-78f5-425d-ab77-8507d378803c-maas-api-xks-ca" pod "66cb95f7-78f5-425d-ab77-8507d378803c" (UID: "66cb95f7-78f5-425d-ab77-8507d378803c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:32:50.789656 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:50.789585 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66cb95f7-78f5-425d-ab77-8507d378803c-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:32:50.789656 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:50.789615 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktq2z\" (UniqueName: \"kubernetes.io/projected/66cb95f7-78f5-425d-ab77-8507d378803c-kube-api-access-ktq2z\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:32:51.478230 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:51.478175 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b" Nov 03 03:32:51.478230 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:51.478211 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b" event={"ID":"66cb95f7-78f5-425d-ab77-8507d378803c","Type":"ContainerDied","Data":"73c76bc838b3110eb8439cf0784cac16d786ef5943014ded7ad3fe375a4c7019"} Nov 03 03:32:51.478451 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:32:51.478252 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73c76bc838b3110eb8439cf0784cac16d786ef5943014ded7ad3fe375a4c7019" Nov 03 03:33:03.923735 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:03.923700 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:04.531249 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.531214 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz"] Nov 03 03:33:04.531646 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.531630 5463 cpu_manager.go:414] "containerMap: removing container" podUID="66cb95f7-78f5-425d-ab77-8507d378803c" containerName="revoke-keys" Nov 03 03:33:04.531701 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.531649 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66cb95f7-78f5-425d-ab77-8507d378803c" containerName="revoke-keys" Nov 03 03:33:04.531781 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.531768 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="66cb95f7-78f5-425d-ab77-8507d378803c" containerName="revoke-keys" Nov 03 03:33:04.535754 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.535731 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.588932 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.588903 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-4e5fe5-openshift-default-dockercfg-2mjdg\"" Nov 03 03:33:04.618552 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.618493 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:04.641758 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.641730 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz"] Nov 03 03:33:04.727780 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.727739 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-workload-socket\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.727972 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.727807 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-envoy\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.727972 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.727895 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-token\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.727972 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.727939 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd7zr\" (UniqueName: \"kubernetes.io/projected/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-kube-api-access-qd7zr\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.727972 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.727965 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-workload-certs\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.728226 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.728045 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-credential-socket\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.728226 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.728068 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-data\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.728226 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.728148 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istiod-ca-cert\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.728226 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.728171 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-podinfo\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.828899 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.828805 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-workload-socket\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.828899 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.828868 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-envoy\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.829151 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.828910 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-token\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.829151 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.828951 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd7zr\" (UniqueName: \"kubernetes.io/projected/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-kube-api-access-qd7zr\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.829151 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.828984 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-workload-certs\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.829151 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.829039 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-credential-socket\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.829151 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.829068 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-data\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.829468 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.829122 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istiod-ca-cert\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.829468 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.829230 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-podinfo\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.829468 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.829241 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-workload-socket\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.829468 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.829297 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:04.829630 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.829492 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-credential-socket\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.829630 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.829583 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-workload-certs\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.829710 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.829677 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-data\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.830061 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.830042 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istiod-ca-cert\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.831318 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.831299 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-envoy\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.832146 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.832127 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-podinfo\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.886285 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.886257 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-token\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:04.886429 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:04.886410 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd7zr\" (UniqueName: \"kubernetes.io/projected/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-kube-api-access-qd7zr\") pod \"e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:05.150008 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:05.149680 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:05.570634 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:33:05.570592 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e7ee83_f0ec_4362_98be_d1cc6bd7cce6.slice/crio-09e83657429c70c5d012d86f0fc9ec830fceb364f7b503c6fc238a7054d6cd8f WatchSource:0}: Error finding container 09e83657429c70c5d012d86f0fc9ec830fceb364f7b503c6fc238a7054d6cd8f: Status 404 returned error can't find the container with id 09e83657429c70c5d012d86f0fc9ec830fceb364f7b503c6fc238a7054d6cd8f Nov 03 03:33:05.573454 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:05.573418 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:33:05.573553 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:05.573489 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:33:05.573553 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:05.573520 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:33:05.573664 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:05.573644 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz"] Nov 03 03:33:06.562375 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:06.562337 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" event={"ID":"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6","Type":"ContainerStarted","Data":"0446e1247f85ef723d0a88f914c5bdfccc1005cd13447781ec25ed494b6b939a"} Nov 03 03:33:06.565032 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:06.562387 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" event={"ID":"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6","Type":"ContainerStarted","Data":"09e83657429c70c5d012d86f0fc9ec830fceb364f7b503c6fc238a7054d6cd8f"} Nov 03 03:33:06.731032 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:06.730979 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" podStartSLOduration=2.73096633 podStartE2EDuration="2.73096633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:33:06.72947325 +0000 UTC m=+6068.413582409" watchObservedRunningTime="2026-11-03 03:33:06.73096633 +0000 UTC m=+6068.415075485" Nov 03 03:33:07.151043 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:07.151008 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:07.156386 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:07.156358 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:07.568245 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:07.568149 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:07.569002 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:07.568980 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:09.632512 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:09.632473 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:09.824495 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:09.824458 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:10.021358 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:10.021324 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:12.846200 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:12.846168 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:13.008299 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:33:13.008264 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:33:13.029228 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:13.029202 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:13.249810 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:13.249771 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd"] Nov 03 03:33:13.253811 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:13.253787 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd" Nov 03 03:33:13.306528 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:13.306492 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-4e5fe5\"" Nov 03 03:33:13.307301 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:13.307276 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd"] Nov 03 03:33:13.316043 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:13.316021 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8ec205-9977-44a1-b242-d78b7bda8dac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd\" (UID: \"ca8ec205-9977-44a1-b242-d78b7bda8dac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd" Nov 03 03:33:13.316144 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:13.316094 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwdkl\" (UniqueName: \"kubernetes.io/projected/ca8ec205-9977-44a1-b242-d78b7bda8dac-kube-api-access-hwdkl\") pod \"payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd\" (UID: \"ca8ec205-9977-44a1-b242-d78b7bda8dac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd" Nov 03 03:33:13.416802 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:13.416767 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwdkl\" (UniqueName: \"kubernetes.io/projected/ca8ec205-9977-44a1-b242-d78b7bda8dac-kube-api-access-hwdkl\") pod \"payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd\" (UID: \"ca8ec205-9977-44a1-b242-d78b7bda8dac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd" Nov 03 03:33:13.416973 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:13.416894 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8ec205-9977-44a1-b242-d78b7bda8dac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd\" (UID: \"ca8ec205-9977-44a1-b242-d78b7bda8dac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd" Nov 03 03:33:13.417620 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:13.417601 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8ec205-9977-44a1-b242-d78b7bda8dac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd\" (UID: \"ca8ec205-9977-44a1-b242-d78b7bda8dac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd" Nov 03 03:33:13.474349 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:13.474320 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwdkl\" (UniqueName: \"kubernetes.io/projected/ca8ec205-9977-44a1-b242-d78b7bda8dac-kube-api-access-hwdkl\") pod \"payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd\" (UID: \"ca8ec205-9977-44a1-b242-d78b7bda8dac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd" Nov 03 03:33:13.568608 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:13.568527 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd" Nov 03 03:33:13.936738 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:33:13.936704 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8ec205_9977_44a1_b242_d78b7bda8dac.slice/crio-87d5ab8277251a3c7159965b92ce2115fc355a300e9abfe4782b9b0201398325 WatchSource:0}: Error finding container 87d5ab8277251a3c7159965b92ce2115fc355a300e9abfe4782b9b0201398325: Status 404 returned error can't find the container with id 87d5ab8277251a3c7159965b92ce2115fc355a300e9abfe4782b9b0201398325 Nov 03 03:33:13.939179 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:13.939128 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd"] Nov 03 03:33:14.611003 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:14.610967 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd" event={"ID":"ca8ec205-9977-44a1-b242-d78b7bda8dac","Type":"ContainerStarted","Data":"54d11c9f251cc9f08b4d2def868c84ab7b865326ace10477444f2f800187083d"} Nov 03 03:33:14.611207 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:14.611018 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd" event={"ID":"ca8ec205-9977-44a1-b242-d78b7bda8dac","Type":"ContainerStarted","Data":"87d5ab8277251a3c7159965b92ce2115fc355a300e9abfe4782b9b0201398325"} Nov 03 03:33:14.611207 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:14.611078 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd" Nov 03 03:33:14.727467 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:14.727417 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd" podStartSLOduration=1.727402311 podStartE2EDuration="1.727402311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:33:14.724937474 +0000 UTC m=+6076.409046629" watchObservedRunningTime="2026-11-03 03:33:14.727402311 +0000 UTC m=+6076.411511463" Nov 03 03:33:15.829825 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:15.829788 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc"] Nov 03 03:33:15.833670 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:15.833652 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" Nov 03 03:33:15.887343 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:15.887313 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc"] Nov 03 03:33:15.934644 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:15.934618 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:33:15.934644 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:15.934621 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-4e5fe5-serving-cert\"" Nov 03 03:33:15.934837 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:15.934677 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:33:15.936609 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:15.936595 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qv59l\"" Nov 03 03:33:15.936674 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:15.936662 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 03:33:15.943570 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:15.943540 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca1bfa3-a951-42ec-8551-37367839f293-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc\" (UID: \"2ca1bfa3-a951-42ec-8551-37367839f293\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" Nov 03 03:33:15.943706 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:15.943578 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pkbp\" (UniqueName: \"kubernetes.io/projected/2ca1bfa3-a951-42ec-8551-37367839f293-kube-api-access-7pkbp\") pod \"maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc\" (UID: \"2ca1bfa3-a951-42ec-8551-37367839f293\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" Nov 03 03:33:15.943706 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:15.943684 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca1bfa3-a951-42ec-8551-37367839f293-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc\" (UID: \"2ca1bfa3-a951-42ec-8551-37367839f293\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" Nov 03 03:33:15.943798 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:15.943714 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca1bfa3-a951-42ec-8551-37367839f293-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc\" (UID: \"2ca1bfa3-a951-42ec-8551-37367839f293\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" Nov 03 03:33:16.040881 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:16.040851 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:16.045050 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:16.045029 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca1bfa3-a951-42ec-8551-37367839f293-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc\" (UID: \"2ca1bfa3-a951-42ec-8551-37367839f293\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" Nov 03 03:33:16.045108 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:16.045059 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca1bfa3-a951-42ec-8551-37367839f293-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc\" (UID: \"2ca1bfa3-a951-42ec-8551-37367839f293\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" Nov 03 03:33:16.045159 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:16.045125 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca1bfa3-a951-42ec-8551-37367839f293-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc\" (UID: \"2ca1bfa3-a951-42ec-8551-37367839f293\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" Nov 03 03:33:16.045159 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:16.045149 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pkbp\" (UniqueName: \"kubernetes.io/projected/2ca1bfa3-a951-42ec-8551-37367839f293-kube-api-access-7pkbp\") pod \"maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc\" (UID: \"2ca1bfa3-a951-42ec-8551-37367839f293\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" Nov 03 03:33:16.045301 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:33:16.045167 5463 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-serving-cert: secret "maas-api-e2e-worker-single-main-4e5fe5-serving-cert" not found Nov 03 03:33:16.045301 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:33:16.045247 5463 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca1bfa3-a951-42ec-8551-37367839f293-maas-api-tls podName:2ca1bfa3-a951-42ec-8551-37367839f293 nodeName:}" failed. No retries permitted until 2026-11-03 03:33:16.545231807 +0000 UTC m=+6078.229340955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2ca1bfa3-a951-42ec-8551-37367839f293-maas-api-tls") pod "maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" (UID: "2ca1bfa3-a951-42ec-8551-37367839f293") : secret "maas-api-e2e-worker-single-main-4e5fe5-serving-cert" not found Nov 03 03:33:16.045780 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:16.045762 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca1bfa3-a951-42ec-8551-37367839f293-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc\" (UID: \"2ca1bfa3-a951-42ec-8551-37367839f293\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" Nov 03 03:33:16.047550 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:16.047531 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca1bfa3-a951-42ec-8551-37367839f293-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc\" (UID: \"2ca1bfa3-a951-42ec-8551-37367839f293\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" Nov 03 03:33:16.103143 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:16.103067 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pkbp\" (UniqueName: \"kubernetes.io/projected/2ca1bfa3-a951-42ec-8551-37367839f293-kube-api-access-7pkbp\") pod \"maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc\" (UID: \"2ca1bfa3-a951-42ec-8551-37367839f293\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" Nov 03 03:33:16.200286 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:16.200247 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:16.354388 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:16.354299 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:16.549814 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:16.549766 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca1bfa3-a951-42ec-8551-37367839f293-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc\" (UID: \"2ca1bfa3-a951-42ec-8551-37367839f293\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" Nov 03 03:33:16.552177 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:16.552151 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca1bfa3-a951-42ec-8551-37367839f293-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc\" (UID: \"2ca1bfa3-a951-42ec-8551-37367839f293\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" Nov 03 03:33:16.747386 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:16.747359 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" Nov 03 03:33:17.115514 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:33:17.115481 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca1bfa3_a951_42ec_8551_37367839f293.slice/crio-b6a73e521e7d5b407a6e8a19c441a2b147d93be93a779948af36ecf9e61cf898 WatchSource:0}: Error finding container b6a73e521e7d5b407a6e8a19c441a2b147d93be93a779948af36ecf9e61cf898: Status 404 returned error can't find the container with id b6a73e521e7d5b407a6e8a19c441a2b147d93be93a779948af36ecf9e61cf898 Nov 03 03:33:17.117162 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:17.117139 5463 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:33:17.118153 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:17.118132 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc"] Nov 03 03:33:17.629811 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:17.629776 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" event={"ID":"2ca1bfa3-a951-42ec-8551-37367839f293","Type":"ContainerStarted","Data":"b6a73e521e7d5b407a6e8a19c441a2b147d93be93a779948af36ecf9e61cf898"} Nov 03 03:33:18.637521 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:18.637479 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" event={"ID":"2ca1bfa3-a951-42ec-8551-37367839f293","Type":"ContainerStarted","Data":"a28c912396430ec9d15c2c7bcede2ba57125650595f5f84ef5f34848eee61745"} Nov 03 03:33:18.637925 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:18.637703 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" Nov 03 03:33:18.758642 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:18.758589 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" podStartSLOduration=2.78343846 podStartE2EDuration="3.758576723s" totalImagesPullingTime="975.138263ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:33:18.756292205 +0000 UTC m=+6080.440401373" watchObservedRunningTime="2026-11-03 03:33:18.758576723 +0000 UTC m=+6080.442685925" Nov 03 03:33:20.977754 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:20.977711 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:22.048903 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:22.048867 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t"] Nov 03 03:33:22.099459 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:22.099431 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-shm5t"] Nov 03 03:33:22.441517 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:22.441486 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbf1ea68-5916-4161-9461-096c8a6dbec8" path="/var/lib/kubelet/pods/dbf1ea68-5916-4161-9461-096c8a6dbec8/volumes" Nov 03 03:33:24.664414 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:24.664371 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" Nov 03 03:33:25.621096 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:25.621068 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd" Nov 03 03:33:25.784012 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:25.783984 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:30.582286 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:30.582205 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:33.747037 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:33.746999 5463 generic.go:356] "Generic (PLEG): container finished" podID="97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" containerID="0446e1247f85ef723d0a88f914c5bdfccc1005cd13447781ec25ed494b6b939a" exitCode=0 Nov 03 03:33:33.747484 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:33.747032 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" event={"ID":"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6","Type":"ContainerDied","Data":"0446e1247f85ef723d0a88f914c5bdfccc1005cd13447781ec25ed494b6b939a"} Nov 03 03:33:33.747484 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:33.747419 5463 scope.go:122] "RemoveContainer" containerID="0446e1247f85ef723d0a88f914c5bdfccc1005cd13447781ec25ed494b6b939a" Nov 03 03:33:33.748532 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:33.748505 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:33:33.748593 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:33.748575 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:33:33.748639 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:33.748610 5463 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 03:33:34.556862 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:34.556800 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:34.778767 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:34.778733 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" event={"ID":"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6","Type":"ContainerStarted","Data":"9526f411a5f32d8cf1d5f8f3ca88816b309e9e70b1138aff780580ef7270c299"} Nov 03 03:33:35.151497 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:35.151465 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:35.160856 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:35.160819 5463 patch_prober.go:29] interesting pod/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Nov 03 03:33:35.161110 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:35.161082 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" podUID="97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Nov 03 03:33:35.192217 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:35.192165 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:35.352517 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:35.352479 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:35.698411 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:35.698378 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:36.168621 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:36.168577 5463 patch_prober.go:29] interesting pod/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Nov 03 03:33:36.169123 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:36.168646 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" podUID="97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Nov 03 03:33:36.246511 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:36.246474 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:37.155216 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:37.151521 5463 patch_prober.go:29] interesting pod/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Nov 03 03:33:37.155216 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:37.151582 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" podUID="97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Nov 03 03:33:37.568378 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:37.568284 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:37.873722 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:37.873608 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:38.161214 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:38.151561 5463 patch_prober.go:29] interesting pod/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Nov 03 03:33:38.161214 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:38.151622 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" podUID="97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Nov 03 03:33:39.151622 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:39.151581 5463 patch_prober.go:29] interesting pod/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Nov 03 03:33:39.152276 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:39.152240 5463 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" podUID="97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Nov 03 03:33:40.154892 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:40.154855 5463 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:40.155840 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:40.155820 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:33:40.496983 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:40.496895 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:43.007299 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:33:43.007256 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:33:45.295352 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:45.295300 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:46.047455 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:46.047421 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc"] Nov 03 03:33:46.097646 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:46.097615 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6zhbc"] Nov 03 03:33:46.442115 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:46.442080 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc" path="/var/lib/kubelet/pods/cc7d0d6a-8eb8-4aa4-a4bb-8b7fdc8194bc/volumes" Nov 03 03:33:50.086800 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:50.086765 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:54.895245 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:54.895173 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:33:59.048713 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:59.048681 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn"] Nov 03 03:33:59.099230 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:59.099200 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4djvn"] Nov 03 03:33:59.672992 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:33:59.672963 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:34:00.442310 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:00.442274 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b277dbf3-c170-44d4-a1a3-1587623db747" path="/var/lib/kubelet/pods/b277dbf3-c170-44d4-a1a3-1587623db747/volumes" Nov 03 03:34:04.499787 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:04.499753 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:34:09.204480 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:09.204225 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:34:13.009447 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:34:13.009387 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:34:13.988087 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:13.988056 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:34:18.802616 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:18.802581 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:34:23.581186 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:23.581152 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:34:28.381407 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:28.381376 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:34:33.180533 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:33.180498 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:34:34.046460 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:34.046419 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l"] Nov 03 03:34:34.098156 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:34.098118 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-77m5l"] Nov 03 03:34:34.441323 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:34.441287 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70dc5d7a-d0a0-499d-a82c-ddc02cfbe426" path="/var/lib/kubelet/pods/70dc5d7a-d0a0-499d-a82c-ddc02cfbe426/volumes" Nov 03 03:34:34.543740 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:34.543708 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:34:35.026047 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:35.025998 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:34:36.486602 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:36.486570 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:34:36.638533 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:36.638497 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:34:37.902965 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:37.902927 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:34:39.601285 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:39.601246 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:34:40.872557 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:40.872526 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:34:41.763938 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:41.763899 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd"] Nov 03 03:34:42.041861 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:42.041769 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:34:42.098026 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:42.097974 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc"] Nov 03 03:34:42.098350 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:42.098281 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" podUID="2ca1bfa3-a951-42ec-8551-37367839f293" containerName="maas-api" containerID="cri-o://a28c912396430ec9d15c2c7bcede2ba57125650595f5f84ef5f34848eee61745" gracePeriod=30 Nov 03 03:34:42.201350 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:42.201317 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:34:42.354161 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:42.354077 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:34:43.006141 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:34:43.006108 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:34:43.350397 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:43.350371 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" Nov 03 03:34:43.425819 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:43.425790 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ca1bfa3-a951-42ec-8551-37367839f293-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca1bfa3-a951-42ec-8551-37367839f293-metrics-certs\") pod \"2ca1bfa3-a951-42ec-8551-37367839f293\" (UID: \"2ca1bfa3-a951-42ec-8551-37367839f293\") " Nov 03 03:34:43.426029 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:43.425870 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ca1bfa3-a951-42ec-8551-37367839f293-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca1bfa3-a951-42ec-8551-37367839f293-maas-api-tls\") pod \"2ca1bfa3-a951-42ec-8551-37367839f293\" (UID: \"2ca1bfa3-a951-42ec-8551-37367839f293\") " Nov 03 03:34:43.426029 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:43.425917 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ca1bfa3-a951-42ec-8551-37367839f293-kube-api-access-7pkbp\" (UniqueName: \"kubernetes.io/projected/2ca1bfa3-a951-42ec-8551-37367839f293-kube-api-access-7pkbp\") pod \"2ca1bfa3-a951-42ec-8551-37367839f293\" (UID: \"2ca1bfa3-a951-42ec-8551-37367839f293\") " Nov 03 03:34:43.426029 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:43.425974 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ca1bfa3-a951-42ec-8551-37367839f293-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca1bfa3-a951-42ec-8551-37367839f293-openshift-service-ca\") pod \"2ca1bfa3-a951-42ec-8551-37367839f293\" (UID: \"2ca1bfa3-a951-42ec-8551-37367839f293\") " Nov 03 03:34:43.426454 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:43.426425 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca1bfa3-a951-42ec-8551-37367839f293-openshift-service-ca" pod "2ca1bfa3-a951-42ec-8551-37367839f293" (UID: "2ca1bfa3-a951-42ec-8551-37367839f293"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:34:43.428075 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:43.428043 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca1bfa3-a951-42ec-8551-37367839f293-kube-api-access-7pkbp" pod "2ca1bfa3-a951-42ec-8551-37367839f293" (UID: "2ca1bfa3-a951-42ec-8551-37367839f293"). InnerVolumeSpecName "kube-api-access-7pkbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:34:43.428075 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:43.428066 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca1bfa3-a951-42ec-8551-37367839f293-metrics-certs" pod "2ca1bfa3-a951-42ec-8551-37367839f293" (UID: "2ca1bfa3-a951-42ec-8551-37367839f293"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:34:43.428299 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:43.428118 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca1bfa3-a951-42ec-8551-37367839f293-maas-api-tls" pod "2ca1bfa3-a951-42ec-8551-37367839f293" (UID: "2ca1bfa3-a951-42ec-8551-37367839f293"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:34:43.527353 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:43.527268 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca1bfa3-a951-42ec-8551-37367839f293-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:34:43.527353 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:43.527296 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca1bfa3-a951-42ec-8551-37367839f293-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:34:43.527353 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:43.527307 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pkbp\" (UniqueName: \"kubernetes.io/projected/2ca1bfa3-a951-42ec-8551-37367839f293-kube-api-access-7pkbp\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:34:43.527353 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:43.527317 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca1bfa3-a951-42ec-8551-37367839f293-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:34:44.197669 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:44.197636 5463 generic.go:356] "Generic (PLEG): container finished" podID="2ca1bfa3-a951-42ec-8551-37367839f293" containerID="a28c912396430ec9d15c2c7bcede2ba57125650595f5f84ef5f34848eee61745" exitCode=0 Nov 03 03:34:44.197847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:44.197677 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" Nov 03 03:34:44.197847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:44.197721 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" event={"ID":"2ca1bfa3-a951-42ec-8551-37367839f293","Type":"ContainerDied","Data":"a28c912396430ec9d15c2c7bcede2ba57125650595f5f84ef5f34848eee61745"} Nov 03 03:34:44.197847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:44.197774 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc" event={"ID":"2ca1bfa3-a951-42ec-8551-37367839f293","Type":"ContainerDied","Data":"b6a73e521e7d5b407a6e8a19c441a2b147d93be93a779948af36ecf9e61cf898"} Nov 03 03:34:44.197847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:44.197801 5463 scope.go:122] "RemoveContainer" containerID="a28c912396430ec9d15c2c7bcede2ba57125650595f5f84ef5f34848eee61745" Nov 03 03:34:44.207972 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:44.207952 5463 scope.go:122] "RemoveContainer" containerID="a28c912396430ec9d15c2c7bcede2ba57125650595f5f84ef5f34848eee61745" Nov 03 03:34:44.208263 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:34:44.208227 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a28c912396430ec9d15c2c7bcede2ba57125650595f5f84ef5f34848eee61745\": container with ID starting with a28c912396430ec9d15c2c7bcede2ba57125650595f5f84ef5f34848eee61745 not found: ID does not exist" containerID="a28c912396430ec9d15c2c7bcede2ba57125650595f5f84ef5f34848eee61745" Nov 03 03:34:44.208358 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:44.208271 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a28c912396430ec9d15c2c7bcede2ba57125650595f5f84ef5f34848eee61745"} err="failed to get container status \"a28c912396430ec9d15c2c7bcede2ba57125650595f5f84ef5f34848eee61745\": rpc error: code = NotFound desc = could not find container \"a28c912396430ec9d15c2c7bcede2ba57125650595f5f84ef5f34848eee61745\": container with ID starting with a28c912396430ec9d15c2c7bcede2ba57125650595f5f84ef5f34848eee61745 not found: ID does not exist" Nov 03 03:34:44.365009 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:44.364977 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc"] Nov 03 03:34:44.415241 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:44.415203 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e5fe5-5687dd4cc5-x7rgc"] Nov 03 03:34:44.441487 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:44.441448 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ca1bfa3-a951-42ec-8551-37367839f293" path="/var/lib/kubelet/pods/2ca1bfa3-a951-42ec-8551-37367839f293/volumes" Nov 03 03:34:45.048979 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:45.048942 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9"] Nov 03 03:34:45.052088 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:45.052060 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k95m9"] Nov 03 03:34:46.333048 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:46.332998 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:34:46.441839 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:46.441806 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bd5d230-0bbe-4410-853d-b149c782dbf9" path="/var/lib/kubelet/pods/9bd5d230-0bbe-4410-853d-b149c782dbf9/volumes" Nov 03 03:34:46.496395 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:46.496365 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:34:46.765164 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:46.765125 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd" podUID="ca8ec205-9977-44a1-b242-d78b7bda8dac" containerName="payload-pre-processing" containerID="cri-o://54d11c9f251cc9f08b4d2def868c84ab7b865326ace10477444f2f800187083d" gracePeriod=25 Nov 03 03:34:55.619165 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:34:55.619128 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd" podUID="ca8ec205-9977-44a1-b242-d78b7bda8dac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:35:05.619283 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:05.619228 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd" podUID="ca8ec205-9977-44a1-b242-d78b7bda8dac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:35:06.433656 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:06.433621 5463 scope.go:122] "RemoveContainer" containerID="27c9c8bae79397504e578c610a96f139a7d3f5dc6f262caac226d30c70892f0b" Nov 03 03:35:06.444012 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:06.443993 5463 scope.go:122] "RemoveContainer" containerID="76ff7011e25eae404fc0f2e50232ab56ccc4020095c38e7bf531ca4151cf5a73" Nov 03 03:35:06.452023 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:06.452004 5463 scope.go:122] "RemoveContainer" containerID="8b75df93541acecad5b41ddcf6621fcb10f9fd866038d35c453b98c3bf7786dc" Nov 03 03:35:07.021289 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:07.021232 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd" Nov 03 03:35:07.053494 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:07.053468 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca8ec205-9977-44a1-b242-d78b7bda8dac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8ec205-9977-44a1-b242-d78b7bda8dac-plugins-config-volume\") pod \"ca8ec205-9977-44a1-b242-d78b7bda8dac\" (UID: \"ca8ec205-9977-44a1-b242-d78b7bda8dac\") " Nov 03 03:35:07.053623 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:07.053529 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca8ec205-9977-44a1-b242-d78b7bda8dac-kube-api-access-hwdkl\" (UniqueName: \"kubernetes.io/projected/ca8ec205-9977-44a1-b242-d78b7bda8dac-kube-api-access-hwdkl\") pod \"ca8ec205-9977-44a1-b242-d78b7bda8dac\" (UID: \"ca8ec205-9977-44a1-b242-d78b7bda8dac\") " Nov 03 03:35:07.053860 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:07.053836 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca8ec205-9977-44a1-b242-d78b7bda8dac-plugins-config-volume" pod "ca8ec205-9977-44a1-b242-d78b7bda8dac" (UID: "ca8ec205-9977-44a1-b242-d78b7bda8dac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:35:07.055599 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:07.055571 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8ec205-9977-44a1-b242-d78b7bda8dac-kube-api-access-hwdkl" pod "ca8ec205-9977-44a1-b242-d78b7bda8dac" (UID: "ca8ec205-9977-44a1-b242-d78b7bda8dac"). InnerVolumeSpecName "kube-api-access-hwdkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:35:07.154836 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:07.154804 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwdkl\" (UniqueName: \"kubernetes.io/projected/ca8ec205-9977-44a1-b242-d78b7bda8dac-kube-api-access-hwdkl\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:35:07.154836 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:07.154832 5463 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8ec205-9977-44a1-b242-d78b7bda8dac-plugins-config-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:35:07.344426 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:07.344347 5463 generic.go:356] "Generic (PLEG): container finished" podID="ca8ec205-9977-44a1-b242-d78b7bda8dac" containerID="54d11c9f251cc9f08b4d2def868c84ab7b865326ace10477444f2f800187083d" exitCode=0 Nov 03 03:35:07.344426 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:07.344407 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd" Nov 03 03:35:07.344804 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:07.344437 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd" event={"ID":"ca8ec205-9977-44a1-b242-d78b7bda8dac","Type":"ContainerDied","Data":"54d11c9f251cc9f08b4d2def868c84ab7b865326ace10477444f2f800187083d"} Nov 03 03:35:07.344804 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:07.344524 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd" event={"ID":"ca8ec205-9977-44a1-b242-d78b7bda8dac","Type":"ContainerDied","Data":"87d5ab8277251a3c7159965b92ce2115fc355a300e9abfe4782b9b0201398325"} Nov 03 03:35:07.344804 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:07.344565 5463 scope.go:122] "RemoveContainer" containerID="54d11c9f251cc9f08b4d2def868c84ab7b865326ace10477444f2f800187083d" Nov 03 03:35:07.353740 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:07.353723 5463 scope.go:122] "RemoveContainer" containerID="54d11c9f251cc9f08b4d2def868c84ab7b865326ace10477444f2f800187083d" Nov 03 03:35:07.354000 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:35:07.353984 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d11c9f251cc9f08b4d2def868c84ab7b865326ace10477444f2f800187083d\": container with ID starting with 54d11c9f251cc9f08b4d2def868c84ab7b865326ace10477444f2f800187083d not found: ID does not exist" containerID="54d11c9f251cc9f08b4d2def868c84ab7b865326ace10477444f2f800187083d" Nov 03 03:35:07.354047 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:07.354007 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d11c9f251cc9f08b4d2def868c84ab7b865326ace10477444f2f800187083d"} err="failed to get container status \"54d11c9f251cc9f08b4d2def868c84ab7b865326ace10477444f2f800187083d\": rpc error: code = NotFound desc = could not find container \"54d11c9f251cc9f08b4d2def868c84ab7b865326ace10477444f2f800187083d\": container with ID starting with 54d11c9f251cc9f08b4d2def868c84ab7b865326ace10477444f2f800187083d not found: ID does not exist" Nov 03 03:35:07.510370 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:07.510339 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd"] Nov 03 03:35:07.560715 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:07.560683 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e5fe5-6cfcbhq6sd"] Nov 03 03:35:08.441183 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:08.441153 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca8ec205-9977-44a1-b242-d78b7bda8dac" path="/var/lib/kubelet/pods/ca8ec205-9977-44a1-b242-d78b7bda8dac/volumes" Nov 03 03:35:13.006444 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:35:13.006407 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:35:16.277627 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:16.277593 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:35:16.450614 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:16.450582 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:35:16.619057 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:16.618974 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:35:18.210798 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:18.210765 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:35:18.575365 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:18.575276 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:35:19.165443 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:19.165405 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz"] Nov 03 03:35:19.165681 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:19.165639 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" podUID="97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" containerName="istio-proxy" containerID="cri-o://9526f411a5f32d8cf1d5f8f3ca88816b309e9e70b1138aff780580ef7270c299" gracePeriod=30 Nov 03 03:35:19.182809 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:19.182783 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:35:20.856978 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:20.856930 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b"] Nov 03 03:35:20.907177 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:20.907145 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5t58b"] Nov 03 03:35:22.313316 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:22.313281 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:35:22.441623 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:22.441587 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66cb95f7-78f5-425d-ab77-8507d378803c" path="/var/lib/kubelet/pods/66cb95f7-78f5-425d-ab77-8507d378803c/volumes" Nov 03 03:35:22.474684 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:22.474644 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:35:24.417682 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.417660 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:35:24.447158 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.447102 5463 generic.go:356] "Generic (PLEG): container finished" podID="97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" containerID="9526f411a5f32d8cf1d5f8f3ca88816b309e9e70b1138aff780580ef7270c299" exitCode=0 Nov 03 03:35:24.447292 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.447153 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" Nov 03 03:35:24.447292 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.447182 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" event={"ID":"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6","Type":"ContainerDied","Data":"9526f411a5f32d8cf1d5f8f3ca88816b309e9e70b1138aff780580ef7270c299"} Nov 03 03:35:24.447292 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.447251 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz" event={"ID":"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6","Type":"ContainerDied","Data":"09e83657429c70c5d012d86f0fc9ec830fceb364f7b503c6fc238a7054d6cd8f"} Nov 03 03:35:24.447292 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.447283 5463 scope.go:122] "RemoveContainer" containerID="9526f411a5f32d8cf1d5f8f3ca88816b309e9e70b1138aff780580ef7270c299" Nov 03 03:35:24.456145 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.456126 5463 scope.go:122] "RemoveContainer" containerID="0446e1247f85ef723d0a88f914c5bdfccc1005cd13447781ec25ed494b6b939a" Nov 03 03:35:24.464597 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.464580 5463 scope.go:122] "RemoveContainer" containerID="9526f411a5f32d8cf1d5f8f3ca88816b309e9e70b1138aff780580ef7270c299" Nov 03 03:35:24.464843 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:35:24.464827 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9526f411a5f32d8cf1d5f8f3ca88816b309e9e70b1138aff780580ef7270c299\": container with ID starting with 9526f411a5f32d8cf1d5f8f3ca88816b309e9e70b1138aff780580ef7270c299 not found: ID does not exist" containerID="9526f411a5f32d8cf1d5f8f3ca88816b309e9e70b1138aff780580ef7270c299" Nov 03 03:35:24.464899 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.464854 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9526f411a5f32d8cf1d5f8f3ca88816b309e9e70b1138aff780580ef7270c299"} err="failed to get container status \"9526f411a5f32d8cf1d5f8f3ca88816b309e9e70b1138aff780580ef7270c299\": rpc error: code = NotFound desc = could not find container \"9526f411a5f32d8cf1d5f8f3ca88816b309e9e70b1138aff780580ef7270c299\": container with ID starting with 9526f411a5f32d8cf1d5f8f3ca88816b309e9e70b1138aff780580ef7270c299 not found: ID does not exist" Nov 03 03:35:24.464899 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.464872 5463 scope.go:122] "RemoveContainer" containerID="0446e1247f85ef723d0a88f914c5bdfccc1005cd13447781ec25ed494b6b939a" Nov 03 03:35:24.465085 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:35:24.465070 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0446e1247f85ef723d0a88f914c5bdfccc1005cd13447781ec25ed494b6b939a\": container with ID starting with 0446e1247f85ef723d0a88f914c5bdfccc1005cd13447781ec25ed494b6b939a not found: ID does not exist" containerID="0446e1247f85ef723d0a88f914c5bdfccc1005cd13447781ec25ed494b6b939a" Nov 03 03:35:24.465123 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.465090 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0446e1247f85ef723d0a88f914c5bdfccc1005cd13447781ec25ed494b6b939a"} err="failed to get container status \"0446e1247f85ef723d0a88f914c5bdfccc1005cd13447781ec25ed494b6b939a\": rpc error: code = NotFound desc = could not find container \"0446e1247f85ef723d0a88f914c5bdfccc1005cd13447781ec25ed494b6b939a\": container with ID starting with 0446e1247f85ef723d0a88f914c5bdfccc1005cd13447781ec25ed494b6b939a not found: ID does not exist" Nov 03 03:35:24.532689 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.532667 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istiod-ca-cert\") pod \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " Nov 03 03:35:24.532809 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.532715 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-workload-socket\") pod \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " Nov 03 03:35:24.532809 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.532753 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-envoy\") pod \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " Nov 03 03:35:24.532809 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.532795 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-podinfo\") pod \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " Nov 03 03:35:24.533002 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.532833 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-data\") pod \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " Nov 03 03:35:24.533002 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.532896 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-workload-certs\") pod \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " Nov 03 03:35:24.533002 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.532921 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-token\" (UniqueName: \"kubernetes.io/projected/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-token\") pod \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " Nov 03 03:35:24.533002 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.532974 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-kube-api-access-qd7zr\" (UniqueName: \"kubernetes.io/projected/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-kube-api-access-qd7zr\") pod \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " Nov 03 03:35:24.533002 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.533000 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-credential-socket\") pod \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\" (UID: \"97e7ee83-f0ec-4362-98be-d1cc6bd7cce6\") " Nov 03 03:35:24.533314 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.533082 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-workload-socket" pod "97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" (UID: "97e7ee83-f0ec-4362-98be-d1cc6bd7cce6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:35:24.533314 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.533131 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istiod-ca-cert" pod "97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" (UID: "97e7ee83-f0ec-4362-98be-d1cc6bd7cce6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:35:24.533442 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.533317 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-credential-socket" pod "97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" (UID: "97e7ee83-f0ec-4362-98be-d1cc6bd7cce6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:35:24.533519 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.533457 5463 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-credential-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:35:24.533519 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.533478 5463 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istiod-ca-cert\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:35:24.533519 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.533493 5463 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-workload-socket\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:35:24.533682 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.533570 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-data" pod "97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" (UID: "97e7ee83-f0ec-4362-98be-d1cc6bd7cce6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:35:24.533682 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.533611 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-workload-certs" pod "97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" (UID: "97e7ee83-f0ec-4362-98be-d1cc6bd7cce6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:35:24.534980 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.534954 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-envoy" pod "97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" (UID: "97e7ee83-f0ec-4362-98be-d1cc6bd7cce6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:35:24.535080 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.535056 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-podinfo" pod "97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" (UID: "97e7ee83-f0ec-4362-98be-d1cc6bd7cce6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:35:24.535136 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.535107 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-kube-api-access-qd7zr" pod "97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" (UID: "97e7ee83-f0ec-4362-98be-d1cc6bd7cce6"). InnerVolumeSpecName "kube-api-access-qd7zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:35:24.535374 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.535348 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-token" pod "97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" (UID: "97e7ee83-f0ec-4362-98be-d1cc6bd7cce6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:35:24.634420 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.634388 5463 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-workload-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:35:24.634420 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.634414 5463 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-token\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:35:24.634420 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.634425 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd7zr\" (UniqueName: \"kubernetes.io/projected/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-kube-api-access-qd7zr\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:35:24.634639 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.634437 5463 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-envoy\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:35:24.634639 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.634446 5463 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-podinfo\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:35:24.634639 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.634454 5463 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6-istio-data\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:35:24.869457 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.869385 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz"] Nov 03 03:35:24.873141 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:24.873113 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4e5fe5-openshift-default-5d4d99544ffr2jz"] Nov 03 03:35:26.441838 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:26.441803 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" path="/var/lib/kubelet/pods/97e7ee83-f0ec-4362-98be-d1cc6bd7cce6/volumes" Nov 03 03:35:36.922778 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:36.922741 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:35:39.932694 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:39.932606 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt"] Nov 03 03:35:39.933362 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:39.932805 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt" podUID="a759ef00-49b6-406f-8a47-255bd11ca523" containerName="manager" containerID="cri-o://450c99b991652262c75bfc18c75941504a830ba5c34c09db43edaac7e3de4c39" gracePeriod=10 Nov 03 03:35:40.048175 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:40.048138 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm"] Nov 03 03:35:40.099582 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:40.099542 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7w5vm"] Nov 03 03:35:40.442658 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:40.442626 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ccee46-86c2-4665-a038-3703352b4979" path="/var/lib/kubelet/pods/65ccee46-86c2-4665-a038-3703352b4979/volumes" Nov 03 03:35:40.548255 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:40.548225 5463 generic.go:356] "Generic (PLEG): container finished" podID="a759ef00-49b6-406f-8a47-255bd11ca523" containerID="450c99b991652262c75bfc18c75941504a830ba5c34c09db43edaac7e3de4c39" exitCode=0 Nov 03 03:35:40.548409 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:40.548303 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt" event={"ID":"a759ef00-49b6-406f-8a47-255bd11ca523","Type":"ContainerDied","Data":"450c99b991652262c75bfc18c75941504a830ba5c34c09db43edaac7e3de4c39"} Nov 03 03:35:40.589025 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:40.589005 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt" Nov 03 03:35:40.706550 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:40.706523 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a759ef00-49b6-406f-8a47-255bd11ca523-kube-api-access-fjn78\" (UniqueName: \"kubernetes.io/projected/a759ef00-49b6-406f-8a47-255bd11ca523-kube-api-access-fjn78\") pod \"a759ef00-49b6-406f-8a47-255bd11ca523\" (UID: \"a759ef00-49b6-406f-8a47-255bd11ca523\") " Nov 03 03:35:40.706697 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:40.706618 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a759ef00-49b6-406f-8a47-255bd11ca523-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a759ef00-49b6-406f-8a47-255bd11ca523-extensions-socket-volume\") pod \"a759ef00-49b6-406f-8a47-255bd11ca523\" (UID: \"a759ef00-49b6-406f-8a47-255bd11ca523\") " Nov 03 03:35:40.707046 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:40.707006 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a759ef00-49b6-406f-8a47-255bd11ca523-extensions-socket-volume" pod "a759ef00-49b6-406f-8a47-255bd11ca523" (UID: "a759ef00-49b6-406f-8a47-255bd11ca523"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:35:40.708807 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:40.708783 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a759ef00-49b6-406f-8a47-255bd11ca523-kube-api-access-fjn78" pod "a759ef00-49b6-406f-8a47-255bd11ca523" (UID: "a759ef00-49b6-406f-8a47-255bd11ca523"). InnerVolumeSpecName "kube-api-access-fjn78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:35:40.808467 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:40.808424 5463 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a759ef00-49b6-406f-8a47-255bd11ca523-extensions-socket-volume\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:35:40.808467 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:40.808462 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjn78\" (UniqueName: \"kubernetes.io/projected/a759ef00-49b6-406f-8a47-255bd11ca523-kube-api-access-fjn78\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:35:41.554957 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:41.554928 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt" Nov 03 03:35:41.555471 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:41.554976 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt" event={"ID":"a759ef00-49b6-406f-8a47-255bd11ca523","Type":"ContainerDied","Data":"594955f5eee04191aec936ac6e83092a87e2fbbc0ee2f2a87f267c52567f57c5"} Nov 03 03:35:41.555471 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:41.555037 5463 scope.go:122] "RemoveContainer" containerID="450c99b991652262c75bfc18c75941504a830ba5c34c09db43edaac7e3de4c39" Nov 03 03:35:41.575017 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:41.574992 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt" podUID="a759ef00-49b6-406f-8a47-255bd11ca523" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:35:41.771504 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:41.771474 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt"] Nov 03 03:35:41.822571 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:41.822493 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sgplt"] Nov 03 03:35:42.441673 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:42.441639 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a759ef00-49b6-406f-8a47-255bd11ca523" path="/var/lib/kubelet/pods/a759ef00-49b6-406f-8a47-255bd11ca523/volumes" Nov 03 03:35:43.006407 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:35:43.006365 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:35:50.945983 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:35:50.945935 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:36:01.046894 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:01.046863 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd"] Nov 03 03:36:01.097326 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:01.097295 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rndzd"] Nov 03 03:36:02.441849 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:02.441815 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f" path="/var/lib/kubelet/pods/8b2c6f15-9f2c-48c8-9fef-1a8eb82ca04f/volumes" Nov 03 03:36:06.485276 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:06.485244 5463 scope.go:122] "RemoveContainer" containerID="f21dde297d208aa292c0a210896be31208dd8fa397165db8d3c0b3b6a812aa43" Nov 03 03:36:06.493954 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:06.493936 5463 scope.go:122] "RemoveContainer" containerID="b695e33d0b8fbf52b09c0d89f91b945b537d9e9a70eb161027bfb713264e6b28" Nov 03 03:36:13.006474 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:36:13.006435 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:36:19.047336 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:19.047304 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv"] Nov 03 03:36:19.097727 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:19.097699 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-lvpzv"] Nov 03 03:36:20.441585 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:20.441552 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d016ecb8-3e97-4ce3-a419-58ad53a2cf7d" path="/var/lib/kubelet/pods/d016ecb8-3e97-4ce3-a419-58ad53a2cf7d/volumes" Nov 03 03:36:37.242487 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:37.242448 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:36:37.393307 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:37.393274 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:36:43.006117 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:36:43.006085 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:36:43.562790 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.562758 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7fl9p"] Nov 03 03:36:43.563253 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.563234 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a759ef00-49b6-406f-8a47-255bd11ca523" containerName="manager" Nov 03 03:36:43.563320 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.563256 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a759ef00-49b6-406f-8a47-255bd11ca523" containerName="manager" Nov 03 03:36:43.563320 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.563287 5463 cpu_manager.go:414] "containerMap: removing container" podUID="97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" containerName="istio-proxy" Nov 03 03:36:43.563320 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.563295 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" containerName="istio-proxy" Nov 03 03:36:43.563320 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.563304 5463 cpu_manager.go:414] "containerMap: removing container" podUID="97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" containerName="istio-proxy" Nov 03 03:36:43.563320 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.563310 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" containerName="istio-proxy" Nov 03 03:36:43.563741 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.563325 5463 cpu_manager.go:414] "containerMap: removing container" podUID="ca8ec205-9977-44a1-b242-d78b7bda8dac" containerName="payload-pre-processing" Nov 03 03:36:43.563741 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.563330 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca8ec205-9977-44a1-b242-d78b7bda8dac" containerName="payload-pre-processing" Nov 03 03:36:43.563741 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.563347 5463 cpu_manager.go:414] "containerMap: removing container" podUID="2ca1bfa3-a951-42ec-8551-37367839f293" containerName="maas-api" Nov 03 03:36:43.563741 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.563353 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ca1bfa3-a951-42ec-8551-37367839f293" containerName="maas-api" Nov 03 03:36:43.563741 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.563424 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ca1bfa3-a951-42ec-8551-37367839f293" containerName="maas-api" Nov 03 03:36:43.563741 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.563434 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a759ef00-49b6-406f-8a47-255bd11ca523" containerName="manager" Nov 03 03:36:43.563741 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.563443 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" containerName="istio-proxy" Nov 03 03:36:43.563741 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.563449 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e7ee83-f0ec-4362-98be-d1cc6bd7cce6" containerName="istio-proxy" Nov 03 03:36:43.563741 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.563456 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca8ec205-9977-44a1-b242-d78b7bda8dac" containerName="payload-pre-processing" Nov 03 03:36:43.566851 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.566825 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7fl9p" Nov 03 03:36:43.611558 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.611534 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7fl9p"] Nov 03 03:36:43.660026 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.660001 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-c5vrr\"" Nov 03 03:36:43.677177 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.677149 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/602b197e-51ff-497e-a430-809c6cd66e0d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7fl9p\" (UID: \"602b197e-51ff-497e-a430-809c6cd66e0d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7fl9p" Nov 03 03:36:43.677309 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.677182 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vg47\" (UniqueName: \"kubernetes.io/projected/602b197e-51ff-497e-a430-809c6cd66e0d-kube-api-access-4vg47\") pod \"kuadrant-operator-controller-manager-678d87766f-7fl9p\" (UID: \"602b197e-51ff-497e-a430-809c6cd66e0d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7fl9p" Nov 03 03:36:43.778249 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.778217 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/602b197e-51ff-497e-a430-809c6cd66e0d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7fl9p\" (UID: \"602b197e-51ff-497e-a430-809c6cd66e0d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7fl9p" Nov 03 03:36:43.778447 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.778255 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vg47\" (UniqueName: \"kubernetes.io/projected/602b197e-51ff-497e-a430-809c6cd66e0d-kube-api-access-4vg47\") pod \"kuadrant-operator-controller-manager-678d87766f-7fl9p\" (UID: \"602b197e-51ff-497e-a430-809c6cd66e0d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7fl9p" Nov 03 03:36:43.778613 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.778591 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/602b197e-51ff-497e-a430-809c6cd66e0d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7fl9p\" (UID: \"602b197e-51ff-497e-a430-809c6cd66e0d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7fl9p" Nov 03 03:36:43.836017 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.835958 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vg47\" (UniqueName: \"kubernetes.io/projected/602b197e-51ff-497e-a430-809c6cd66e0d-kube-api-access-4vg47\") pod \"kuadrant-operator-controller-manager-678d87766f-7fl9p\" (UID: \"602b197e-51ff-497e-a430-809c6cd66e0d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7fl9p" Nov 03 03:36:43.878945 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:43.878914 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7fl9p" Nov 03 03:36:44.341483 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:36:44.341449 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602b197e_51ff_497e_a430_809c6cd66e0d.slice/crio-ed5bc636c42089d4ea6c3b5e0384922ca9e12eb736ccb9fc9740e552fc38640f WatchSource:0}: Error finding container ed5bc636c42089d4ea6c3b5e0384922ca9e12eb736ccb9fc9740e552fc38640f: Status 404 returned error can't find the container with id ed5bc636c42089d4ea6c3b5e0384922ca9e12eb736ccb9fc9740e552fc38640f Nov 03 03:36:44.346254 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:44.344351 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7fl9p"] Nov 03 03:36:44.904500 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:44.904465 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7fl9p" event={"ID":"602b197e-51ff-497e-a430-809c6cd66e0d","Type":"ContainerStarted","Data":"2ccb971a9a363a1d5a0411ecc3c9d53a60c8f006f4fa91a6b838e98045799f98"} Nov 03 03:36:44.904698 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:44.904525 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7fl9p" event={"ID":"602b197e-51ff-497e-a430-809c6cd66e0d","Type":"ContainerStarted","Data":"ed5bc636c42089d4ea6c3b5e0384922ca9e12eb736ccb9fc9740e552fc38640f"} Nov 03 03:36:44.904698 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:44.904629 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7fl9p" Nov 03 03:36:45.123621 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:45.123569 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7fl9p" podStartSLOduration=2.1235531930000002 podStartE2EDuration="2.123553193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:36:45.072038658 +0000 UTC m=+6286.756147815" watchObservedRunningTime="2026-11-03 03:36:45.123553193 +0000 UTC m=+6286.807662347" Nov 03 03:36:55.912069 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:36:55.912040 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7fl9p" Nov 03 03:37:03.529400 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:03.529374 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 03:37:03.537264 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:03.537243 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 03:37:06.531259 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:06.531230 5463 scope.go:122] "RemoveContainer" containerID="0bd4c369ccdcb331531c5b5a7f749ca9c17580fff3a6e5aa29fba5bcb59d5374" Nov 03 03:37:06.540370 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:06.540349 5463 scope.go:122] "RemoveContainer" containerID="c4d69b2cb94c906db6cd18973e66ef7532dead56f38fda5a4c81b886909b3b3c" Nov 03 03:37:11.436483 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:11.436406 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g"] Nov 03 03:37:11.443941 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:11.443923 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g" Nov 03 03:37:11.492776 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:11.492747 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g"] Nov 03 03:37:11.541057 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:11.541028 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:37:11.541183 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:11.541072 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:37:11.541183 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:11.541117 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:37:11.546709 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:11.546695 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:37:11.556405 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:11.556381 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9183e2d0-6e9b-49cd-bf37-695924c25cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-t7v4g\" (UID: \"9183e2d0-6e9b-49cd-bf37-695924c25cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g" Nov 03 03:37:11.556676 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:11.556645 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9183e2d0-6e9b-49cd-bf37-695924c25cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-t7v4g\" (UID: \"9183e2d0-6e9b-49cd-bf37-695924c25cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g" Nov 03 03:37:11.556772 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:11.556747 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68n69\" (UniqueName: \"kubernetes.io/projected/9183e2d0-6e9b-49cd-bf37-695924c25cb1-kube-api-access-68n69\") pod \"maas-api-revoke-sub-d576e68b-t7v4g\" (UID: \"9183e2d0-6e9b-49cd-bf37-695924c25cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g" Nov 03 03:37:11.658261 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:11.658223 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9183e2d0-6e9b-49cd-bf37-695924c25cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-t7v4g\" (UID: \"9183e2d0-6e9b-49cd-bf37-695924c25cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g" Nov 03 03:37:11.658416 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:11.658289 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68n69\" (UniqueName: \"kubernetes.io/projected/9183e2d0-6e9b-49cd-bf37-695924c25cb1-kube-api-access-68n69\") pod \"maas-api-revoke-sub-d576e68b-t7v4g\" (UID: \"9183e2d0-6e9b-49cd-bf37-695924c25cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g" Nov 03 03:37:11.658416 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:11.658334 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9183e2d0-6e9b-49cd-bf37-695924c25cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-t7v4g\" (UID: \"9183e2d0-6e9b-49cd-bf37-695924c25cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g" Nov 03 03:37:11.658886 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:11.658862 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9183e2d0-6e9b-49cd-bf37-695924c25cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-t7v4g\" (UID: \"9183e2d0-6e9b-49cd-bf37-695924c25cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g" Nov 03 03:37:11.660645 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:11.660622 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9183e2d0-6e9b-49cd-bf37-695924c25cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-t7v4g\" (UID: \"9183e2d0-6e9b-49cd-bf37-695924c25cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g" Nov 03 03:37:11.715543 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:11.715483 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68n69\" (UniqueName: \"kubernetes.io/projected/9183e2d0-6e9b-49cd-bf37-695924c25cb1-kube-api-access-68n69\") pod \"maas-api-revoke-sub-d576e68b-t7v4g\" (UID: \"9183e2d0-6e9b-49cd-bf37-695924c25cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g" Nov 03 03:37:11.759152 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:11.759128 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g" Nov 03 03:37:12.122185 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:37:12.122154 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9183e2d0_6e9b_49cd_bf37_695924c25cb1.slice/crio-13bafb7156feccac7f55303b2f168758005779066a969dc009fcd5f1c506571c WatchSource:0}: Error finding container 13bafb7156feccac7f55303b2f168758005779066a969dc009fcd5f1c506571c: Status 404 returned error can't find the container with id 13bafb7156feccac7f55303b2f168758005779066a969dc009fcd5f1c506571c Nov 03 03:37:12.124177 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:12.124157 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g"] Nov 03 03:37:13.006261 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:37:13.006216 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:37:13.062275 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:13.062242 5463 generic.go:356] "Generic (PLEG): container finished" podID="9183e2d0-6e9b-49cd-bf37-695924c25cb1" containerID="c2230e959fa7a1396d5f14beddee2e2939fa6c8a50e53c1127a0b1bf5143b90a" exitCode=0 Nov 03 03:37:13.062453 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:13.062431 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g" event={"ID":"9183e2d0-6e9b-49cd-bf37-695924c25cb1","Type":"ContainerDied","Data":"c2230e959fa7a1396d5f14beddee2e2939fa6c8a50e53c1127a0b1bf5143b90a"} Nov 03 03:37:13.062570 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:13.062487 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g" event={"ID":"9183e2d0-6e9b-49cd-bf37-695924c25cb1","Type":"ContainerStarted","Data":"13bafb7156feccac7f55303b2f168758005779066a969dc009fcd5f1c506571c"} Nov 03 03:37:14.202769 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:14.202743 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g" Nov 03 03:37:14.281312 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:14.281289 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9183e2d0-6e9b-49cd-bf37-695924c25cb1-kube-api-access-68n69\" (UniqueName: \"kubernetes.io/projected/9183e2d0-6e9b-49cd-bf37-695924c25cb1-kube-api-access-68n69\") pod \"9183e2d0-6e9b-49cd-bf37-695924c25cb1\" (UID: \"9183e2d0-6e9b-49cd-bf37-695924c25cb1\") " Nov 03 03:37:14.281455 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:14.281353 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9183e2d0-6e9b-49cd-bf37-695924c25cb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9183e2d0-6e9b-49cd-bf37-695924c25cb1-maas-api-openshift-ca\") pod \"9183e2d0-6e9b-49cd-bf37-695924c25cb1\" (UID: \"9183e2d0-6e9b-49cd-bf37-695924c25cb1\") " Nov 03 03:37:14.281455 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:14.281447 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9183e2d0-6e9b-49cd-bf37-695924c25cb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9183e2d0-6e9b-49cd-bf37-695924c25cb1-maas-api-xks-ca\") pod \"9183e2d0-6e9b-49cd-bf37-695924c25cb1\" (UID: \"9183e2d0-6e9b-49cd-bf37-695924c25cb1\") " Nov 03 03:37:14.281646 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:14.281630 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9183e2d0-6e9b-49cd-bf37-695924c25cb1-maas-api-openshift-ca" pod "9183e2d0-6e9b-49cd-bf37-695924c25cb1" (UID: "9183e2d0-6e9b-49cd-bf37-695924c25cb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:37:14.281748 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:14.281722 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9183e2d0-6e9b-49cd-bf37-695924c25cb1-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:37:14.283511 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:14.283494 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9183e2d0-6e9b-49cd-bf37-695924c25cb1-maas-api-xks-ca" pod "9183e2d0-6e9b-49cd-bf37-695924c25cb1" (UID: "9183e2d0-6e9b-49cd-bf37-695924c25cb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:37:14.283586 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:14.283519 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9183e2d0-6e9b-49cd-bf37-695924c25cb1-kube-api-access-68n69" pod "9183e2d0-6e9b-49cd-bf37-695924c25cb1" (UID: "9183e2d0-6e9b-49cd-bf37-695924c25cb1"). InnerVolumeSpecName "kube-api-access-68n69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:37:14.382812 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:14.382742 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9183e2d0-6e9b-49cd-bf37-695924c25cb1-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:37:14.382812 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:14.382775 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68n69\" (UniqueName: \"kubernetes.io/projected/9183e2d0-6e9b-49cd-bf37-695924c25cb1-kube-api-access-68n69\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:37:15.075904 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:15.075875 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g" Nov 03 03:37:15.076087 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:15.075875 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g" event={"ID":"9183e2d0-6e9b-49cd-bf37-695924c25cb1","Type":"ContainerDied","Data":"13bafb7156feccac7f55303b2f168758005779066a969dc009fcd5f1c506571c"} Nov 03 03:37:15.076087 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:15.076002 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13bafb7156feccac7f55303b2f168758005779066a969dc009fcd5f1c506571c" Nov 03 03:37:23.049519 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:23.049482 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k"] Nov 03 03:37:23.099754 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:23.099722 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fph7k"] Nov 03 03:37:24.441930 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:24.441898 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c325533f-3347-4531-b023-1fdd3bf2eafc" path="/var/lib/kubelet/pods/c325533f-3347-4531-b023-1fdd3bf2eafc/volumes" Nov 03 03:37:33.024847 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.024814 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb"] Nov 03 03:37:33.025345 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.025326 5463 cpu_manager.go:414] "containerMap: removing container" podUID="9183e2d0-6e9b-49cd-bf37-695924c25cb1" containerName="revoke-keys" Nov 03 03:37:33.025406 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.025349 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9183e2d0-6e9b-49cd-bf37-695924c25cb1" containerName="revoke-keys" Nov 03 03:37:33.025501 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.025490 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="9183e2d0-6e9b-49cd-bf37-695924c25cb1" containerName="revoke-keys" Nov 03 03:37:33.028870 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.028855 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb" Nov 03 03:37:33.037605 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.037577 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da2e572-801b-4b48-9236-e06a3a0e7adc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-cqspb\" (UID: \"0da2e572-801b-4b48-9236-e06a3a0e7adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb" Nov 03 03:37:33.037744 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.037621 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da2e572-801b-4b48-9236-e06a3a0e7adc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-cqspb\" (UID: \"0da2e572-801b-4b48-9236-e06a3a0e7adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb" Nov 03 03:37:33.037744 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.037708 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grkjg\" (UniqueName: \"kubernetes.io/projected/0da2e572-801b-4b48-9236-e06a3a0e7adc-kube-api-access-grkjg\") pod \"maas-api-revoke-sub-d2bc93bc-cqspb\" (UID: \"0da2e572-801b-4b48-9236-e06a3a0e7adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb" Nov 03 03:37:33.081863 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.081826 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb"] Nov 03 03:37:33.130167 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.130131 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:37:33.130167 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.130158 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:37:33.130391 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.130185 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:37:33.131835 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.131816 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sb4qh\"" Nov 03 03:37:33.138356 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.138334 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da2e572-801b-4b48-9236-e06a3a0e7adc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-cqspb\" (UID: \"0da2e572-801b-4b48-9236-e06a3a0e7adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb" Nov 03 03:37:33.138412 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.138380 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da2e572-801b-4b48-9236-e06a3a0e7adc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-cqspb\" (UID: \"0da2e572-801b-4b48-9236-e06a3a0e7adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb" Nov 03 03:37:33.138457 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.138410 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grkjg\" (UniqueName: \"kubernetes.io/projected/0da2e572-801b-4b48-9236-e06a3a0e7adc-kube-api-access-grkjg\") pod \"maas-api-revoke-sub-d2bc93bc-cqspb\" (UID: \"0da2e572-801b-4b48-9236-e06a3a0e7adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb" Nov 03 03:37:33.139088 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.139067 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da2e572-801b-4b48-9236-e06a3a0e7adc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-cqspb\" (UID: \"0da2e572-801b-4b48-9236-e06a3a0e7adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb" Nov 03 03:37:33.140737 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.140713 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da2e572-801b-4b48-9236-e06a3a0e7adc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-cqspb\" (UID: \"0da2e572-801b-4b48-9236-e06a3a0e7adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb" Nov 03 03:37:33.196111 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.196087 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grkjg\" (UniqueName: \"kubernetes.io/projected/0da2e572-801b-4b48-9236-e06a3a0e7adc-kube-api-access-grkjg\") pod \"maas-api-revoke-sub-d2bc93bc-cqspb\" (UID: \"0da2e572-801b-4b48-9236-e06a3a0e7adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb" Nov 03 03:37:33.342155 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.342077 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb" Nov 03 03:37:33.726850 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:37:33.726817 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da2e572_801b_4b48_9236_e06a3a0e7adc.slice/crio-f19704ce0fcffe2b3670ffeaaddbd1c56e3f63c8adc2ca5f5138cdd5c4006fd6 WatchSource:0}: Error finding container f19704ce0fcffe2b3670ffeaaddbd1c56e3f63c8adc2ca5f5138cdd5c4006fd6: Status 404 returned error can't find the container with id f19704ce0fcffe2b3670ffeaaddbd1c56e3f63c8adc2ca5f5138cdd5c4006fd6 Nov 03 03:37:33.728840 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:33.728819 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb"] Nov 03 03:37:34.048963 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:34.048926 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz"] Nov 03 03:37:34.099113 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:34.099088 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-56mgz"] Nov 03 03:37:34.237076 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:34.237046 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb" event={"ID":"0da2e572-801b-4b48-9236-e06a3a0e7adc","Type":"ContainerStarted","Data":"4f817b94b44d9418b94296be627af689f5ab3ca407b70783ec03e5fd38d16f75"} Nov 03 03:37:34.237300 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:34.237100 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb" event={"ID":"0da2e572-801b-4b48-9236-e06a3a0e7adc","Type":"ContainerStarted","Data":"f19704ce0fcffe2b3670ffeaaddbd1c56e3f63c8adc2ca5f5138cdd5c4006fd6"} Nov 03 03:37:34.351892 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:34.351801 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb" podStartSLOduration=2.351787329 podStartE2EDuration="2.351787329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:37:34.350631084 +0000 UTC m=+6336.034740240" watchObservedRunningTime="2026-11-03 03:37:34.351787329 +0000 UTC m=+6336.035896524" Nov 03 03:37:34.442510 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:34.442478 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04cf8753-4e9d-443d-86ba-9f9d49b7affd" path="/var/lib/kubelet/pods/04cf8753-4e9d-443d-86ba-9f9d49b7affd/volumes" Nov 03 03:37:35.245800 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:35.245766 5463 generic.go:356] "Generic (PLEG): container finished" podID="0da2e572-801b-4b48-9236-e06a3a0e7adc" containerID="4f817b94b44d9418b94296be627af689f5ab3ca407b70783ec03e5fd38d16f75" exitCode=0 Nov 03 03:37:35.246330 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:35.245850 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb" event={"ID":"0da2e572-801b-4b48-9236-e06a3a0e7adc","Type":"ContainerDied","Data":"4f817b94b44d9418b94296be627af689f5ab3ca407b70783ec03e5fd38d16f75"} Nov 03 03:37:36.382703 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:36.382669 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb" Nov 03 03:37:36.471447 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:36.471417 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0da2e572-801b-4b48-9236-e06a3a0e7adc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da2e572-801b-4b48-9236-e06a3a0e7adc-maas-api-xks-ca\") pod \"0da2e572-801b-4b48-9236-e06a3a0e7adc\" (UID: \"0da2e572-801b-4b48-9236-e06a3a0e7adc\") " Nov 03 03:37:36.471604 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:36.471450 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da2e572-801b-4b48-9236-e06a3a0e7adc-kube-api-access-grkjg\" (UniqueName: \"kubernetes.io/projected/0da2e572-801b-4b48-9236-e06a3a0e7adc-kube-api-access-grkjg\") pod \"0da2e572-801b-4b48-9236-e06a3a0e7adc\" (UID: \"0da2e572-801b-4b48-9236-e06a3a0e7adc\") " Nov 03 03:37:36.471604 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:36.471529 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da2e572-801b-4b48-9236-e06a3a0e7adc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da2e572-801b-4b48-9236-e06a3a0e7adc-maas-api-openshift-ca\") pod \"0da2e572-801b-4b48-9236-e06a3a0e7adc\" (UID: \"0da2e572-801b-4b48-9236-e06a3a0e7adc\") " Nov 03 03:37:36.471914 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:36.471891 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da2e572-801b-4b48-9236-e06a3a0e7adc-maas-api-openshift-ca" pod "0da2e572-801b-4b48-9236-e06a3a0e7adc" (UID: "0da2e572-801b-4b48-9236-e06a3a0e7adc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:37:36.473475 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:36.473454 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da2e572-801b-4b48-9236-e06a3a0e7adc-maas-api-xks-ca" pod "0da2e572-801b-4b48-9236-e06a3a0e7adc" (UID: "0da2e572-801b-4b48-9236-e06a3a0e7adc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:37:36.473568 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:36.473544 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da2e572-801b-4b48-9236-e06a3a0e7adc-kube-api-access-grkjg" pod "0da2e572-801b-4b48-9236-e06a3a0e7adc" (UID: "0da2e572-801b-4b48-9236-e06a3a0e7adc"). InnerVolumeSpecName "kube-api-access-grkjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:37:36.572231 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:36.572136 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da2e572-801b-4b48-9236-e06a3a0e7adc-maas-api-openshift-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:37:36.572231 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:36.572165 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da2e572-801b-4b48-9236-e06a3a0e7adc-maas-api-xks-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:37:36.572231 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:36.572175 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grkjg\" (UniqueName: \"kubernetes.io/projected/0da2e572-801b-4b48-9236-e06a3a0e7adc-kube-api-access-grkjg\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:37:37.260398 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:37.260366 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb" event={"ID":"0da2e572-801b-4b48-9236-e06a3a0e7adc","Type":"ContainerDied","Data":"f19704ce0fcffe2b3670ffeaaddbd1c56e3f63c8adc2ca5f5138cdd5c4006fd6"} Nov 03 03:37:37.260577 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:37.260413 5463 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f19704ce0fcffe2b3670ffeaaddbd1c56e3f63c8adc2ca5f5138cdd5c4006fd6" Nov 03 03:37:37.260577 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:37.260419 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb" Nov 03 03:37:37.620549 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:37.620469 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:37:43.006842 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:37:43.006800 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:37:46.243031 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.242998 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g"] Nov 03 03:37:46.243467 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.243452 5463 cpu_manager.go:414] "containerMap: removing container" podUID="0da2e572-801b-4b48-9236-e06a3a0e7adc" containerName="revoke-keys" Nov 03 03:37:46.243521 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.243469 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da2e572-801b-4b48-9236-e06a3a0e7adc" containerName="revoke-keys" Nov 03 03:37:46.243565 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.243549 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da2e572-801b-4b48-9236-e06a3a0e7adc" containerName="revoke-keys" Nov 03 03:37:46.246846 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.246828 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" Nov 03 03:37:46.255732 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.255715 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52"] Nov 03 03:37:46.259204 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.259178 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52" Nov 03 03:37:46.304020 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.303988 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:37:46.304358 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.304333 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:37:46.304491 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.304361 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qv59l\"" Nov 03 03:37:46.350446 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.350400 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 03:37:46.350688 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.350469 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 03:37:46.352573 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.352546 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g"] Nov 03 03:37:46.354318 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.354296 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52"] Nov 03 03:37:46.362731 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.362699 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-maas-api-tls\") pod \"maas-api-f9b4c8d5c-6bk7g\" (UID: \"a35d11ca-ac2d-4c68-8495-df9f7ed11c96\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" Nov 03 03:37:46.362835 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.362813 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcjl4\" (UniqueName: \"kubernetes.io/projected/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-kube-api-access-xcjl4\") pod \"maas-api-f9b4c8d5c-6bk7g\" (UID: \"a35d11ca-ac2d-4c68-8495-df9f7ed11c96\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" Nov 03 03:37:46.362899 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.362879 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xprzs\" (UniqueName: \"kubernetes.io/projected/08f91ace-969b-4cd9-b322-d754e287385e-kube-api-access-xprzs\") pod \"maas-api-5bfdcdb597-j4c52\" (UID: \"08f91ace-969b-4cd9-b322-d754e287385e\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52" Nov 03 03:37:46.362950 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.362916 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08f91ace-969b-4cd9-b322-d754e287385e-metrics-certs\") pod \"maas-api-5bfdcdb597-j4c52\" (UID: \"08f91ace-969b-4cd9-b322-d754e287385e\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52" Nov 03 03:37:46.363053 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.363037 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08f91ace-969b-4cd9-b322-d754e287385e-maas-api-tls\") pod \"maas-api-5bfdcdb597-j4c52\" (UID: \"08f91ace-969b-4cd9-b322-d754e287385e\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52" Nov 03 03:37:46.363121 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.363100 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f91ace-969b-4cd9-b322-d754e287385e-openshift-service-ca\") pod \"maas-api-5bfdcdb597-j4c52\" (UID: \"08f91ace-969b-4cd9-b322-d754e287385e\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52" Nov 03 03:37:46.363164 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.363155 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-openshift-service-ca\") pod \"maas-api-f9b4c8d5c-6bk7g\" (UID: \"a35d11ca-ac2d-4c68-8495-df9f7ed11c96\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" Nov 03 03:37:46.363293 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.363274 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-metrics-certs\") pod \"maas-api-f9b4c8d5c-6bk7g\" (UID: \"a35d11ca-ac2d-4c68-8495-df9f7ed11c96\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" Nov 03 03:37:46.464036 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.463999 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-maas-api-tls\") pod \"maas-api-f9b4c8d5c-6bk7g\" (UID: \"a35d11ca-ac2d-4c68-8495-df9f7ed11c96\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" Nov 03 03:37:46.464036 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.464039 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcjl4\" (UniqueName: \"kubernetes.io/projected/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-kube-api-access-xcjl4\") pod \"maas-api-f9b4c8d5c-6bk7g\" (UID: \"a35d11ca-ac2d-4c68-8495-df9f7ed11c96\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" Nov 03 03:37:46.464323 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.464219 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xprzs\" (UniqueName: \"kubernetes.io/projected/08f91ace-969b-4cd9-b322-d754e287385e-kube-api-access-xprzs\") pod \"maas-api-5bfdcdb597-j4c52\" (UID: \"08f91ace-969b-4cd9-b322-d754e287385e\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52" Nov 03 03:37:46.464323 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.464266 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08f91ace-969b-4cd9-b322-d754e287385e-metrics-certs\") pod \"maas-api-5bfdcdb597-j4c52\" (UID: \"08f91ace-969b-4cd9-b322-d754e287385e\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52" Nov 03 03:37:46.464450 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.464335 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08f91ace-969b-4cd9-b322-d754e287385e-maas-api-tls\") pod \"maas-api-5bfdcdb597-j4c52\" (UID: \"08f91ace-969b-4cd9-b322-d754e287385e\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52" Nov 03 03:37:46.464450 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.464367 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f91ace-969b-4cd9-b322-d754e287385e-openshift-service-ca\") pod \"maas-api-5bfdcdb597-j4c52\" (UID: \"08f91ace-969b-4cd9-b322-d754e287385e\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52" Nov 03 03:37:46.464450 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.464411 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-openshift-service-ca\") pod \"maas-api-f9b4c8d5c-6bk7g\" (UID: \"a35d11ca-ac2d-4c68-8495-df9f7ed11c96\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" Nov 03 03:37:46.464617 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.464464 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-metrics-certs\") pod \"maas-api-f9b4c8d5c-6bk7g\" (UID: \"a35d11ca-ac2d-4c68-8495-df9f7ed11c96\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" Nov 03 03:37:46.465334 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.465303 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-openshift-service-ca\") pod \"maas-api-f9b4c8d5c-6bk7g\" (UID: \"a35d11ca-ac2d-4c68-8495-df9f7ed11c96\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" Nov 03 03:37:46.465471 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.465407 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f91ace-969b-4cd9-b322-d754e287385e-openshift-service-ca\") pod \"maas-api-5bfdcdb597-j4c52\" (UID: \"08f91ace-969b-4cd9-b322-d754e287385e\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52" Nov 03 03:37:46.467217 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.467167 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-maas-api-tls\") pod \"maas-api-f9b4c8d5c-6bk7g\" (UID: \"a35d11ca-ac2d-4c68-8495-df9f7ed11c96\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" Nov 03 03:37:46.467505 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.467473 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-metrics-certs\") pod \"maas-api-f9b4c8d5c-6bk7g\" (UID: \"a35d11ca-ac2d-4c68-8495-df9f7ed11c96\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" Nov 03 03:37:46.467588 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.467473 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08f91ace-969b-4cd9-b322-d754e287385e-metrics-certs\") pod \"maas-api-5bfdcdb597-j4c52\" (UID: \"08f91ace-969b-4cd9-b322-d754e287385e\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52" Nov 03 03:37:46.467588 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.467513 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08f91ace-969b-4cd9-b322-d754e287385e-maas-api-tls\") pod \"maas-api-5bfdcdb597-j4c52\" (UID: \"08f91ace-969b-4cd9-b322-d754e287385e\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52" Nov 03 03:37:46.521073 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.521006 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcjl4\" (UniqueName: \"kubernetes.io/projected/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-kube-api-access-xcjl4\") pod \"maas-api-f9b4c8d5c-6bk7g\" (UID: \"a35d11ca-ac2d-4c68-8495-df9f7ed11c96\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" Nov 03 03:37:46.521226 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.521103 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xprzs\" (UniqueName: \"kubernetes.io/projected/08f91ace-969b-4cd9-b322-d754e287385e-kube-api-access-xprzs\") pod \"maas-api-5bfdcdb597-j4c52\" (UID: \"08f91ace-969b-4cd9-b322-d754e287385e\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52" Nov 03 03:37:46.563182 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.563152 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" Nov 03 03:37:46.574138 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.574114 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52" Nov 03 03:37:46.941401 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:37:46.941369 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda35d11ca_ac2d_4c68_8495_df9f7ed11c96.slice/crio-4336ac27590ac82cbff01ba166d6844205b27124244362d76d6256b5a0cab54d WatchSource:0}: Error finding container 4336ac27590ac82cbff01ba166d6844205b27124244362d76d6256b5a0cab54d: Status 404 returned error can't find the container with id 4336ac27590ac82cbff01ba166d6844205b27124244362d76d6256b5a0cab54d Nov 03 03:37:46.942598 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.942572 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g"] Nov 03 03:37:46.980851 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:37:46.980822 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f91ace_969b_4cd9_b322_d754e287385e.slice/crio-f8146c01a117798ad436b3384886f7766c2ff8943850f824eb2be5c243dc9db0 WatchSource:0}: Error finding container f8146c01a117798ad436b3384886f7766c2ff8943850f824eb2be5c243dc9db0: Status 404 returned error can't find the container with id f8146c01a117798ad436b3384886f7766c2ff8943850f824eb2be5c243dc9db0 Nov 03 03:37:46.983349 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:46.983329 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52"] Nov 03 03:37:47.324934 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:47.324841 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" event={"ID":"a35d11ca-ac2d-4c68-8495-df9f7ed11c96","Type":"ContainerStarted","Data":"4336ac27590ac82cbff01ba166d6844205b27124244362d76d6256b5a0cab54d"} Nov 03 03:37:47.326096 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:47.326072 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52" event={"ID":"08f91ace-969b-4cd9-b322-d754e287385e","Type":"ContainerStarted","Data":"f8146c01a117798ad436b3384886f7766c2ff8943850f824eb2be5c243dc9db0"} Nov 03 03:37:48.334277 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:48.334156 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" event={"ID":"a35d11ca-ac2d-4c68-8495-df9f7ed11c96","Type":"ContainerStarted","Data":"a67e09b23cf0161b368ee50d33eddf3d3e4492895ffed2876304ded6fd663e45"} Nov 03 03:37:48.334277 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:48.334253 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" Nov 03 03:37:48.335880 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:48.335855 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52" event={"ID":"08f91ace-969b-4cd9-b322-d754e287385e","Type":"ContainerStarted","Data":"10796e2822670bc001aa910414fcac53aed52be97584ce454ce2a21c04aa80ba"} Nov 03 03:37:48.336006 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:48.335933 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52" Nov 03 03:37:48.451713 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:48.451661 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" podStartSLOduration=1.477376249 podStartE2EDuration="2.451646951s" totalImagesPullingTime="974.270702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:37:48.449362916 +0000 UTC m=+6350.133472071" watchObservedRunningTime="2026-11-03 03:37:48.451646951 +0000 UTC m=+6350.135756106" Nov 03 03:37:48.622463 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:48.622344 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52" podStartSLOduration=1.68665382 podStartE2EDuration="2.622328469s" totalImagesPullingTime="935.674649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:37:48.620622909 +0000 UTC m=+6350.304732064" watchObservedRunningTime="2026-11-03 03:37:48.622328469 +0000 UTC m=+6350.306437624" Nov 03 03:37:54.346289 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:54.346248 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-j4c52" Nov 03 03:37:54.346899 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:54.346639 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" Nov 03 03:37:54.513516 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:54.513479 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g"] Nov 03 03:37:54.513738 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:54.513710 5463 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" podUID="a35d11ca-ac2d-4c68-8495-df9f7ed11c96" containerName="maas-api" containerID="cri-o://a67e09b23cf0161b368ee50d33eddf3d3e4492895ffed2876304ded6fd663e45" gracePeriod=30 Nov 03 03:37:54.767651 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:54.767621 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" Nov 03 03:37:54.845611 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:54.845583 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-openshift-service-ca\") pod \"a35d11ca-ac2d-4c68-8495-df9f7ed11c96\" (UID: \"a35d11ca-ac2d-4c68-8495-df9f7ed11c96\") " Nov 03 03:37:54.845748 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:54.845674 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-kube-api-access-xcjl4\" (UniqueName: \"kubernetes.io/projected/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-kube-api-access-xcjl4\") pod \"a35d11ca-ac2d-4c68-8495-df9f7ed11c96\" (UID: \"a35d11ca-ac2d-4c68-8495-df9f7ed11c96\") " Nov 03 03:37:54.845748 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:54.845714 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-metrics-certs\") pod \"a35d11ca-ac2d-4c68-8495-df9f7ed11c96\" (UID: \"a35d11ca-ac2d-4c68-8495-df9f7ed11c96\") " Nov 03 03:37:54.845845 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:54.845767 5463 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-maas-api-tls\") pod \"a35d11ca-ac2d-4c68-8495-df9f7ed11c96\" (UID: \"a35d11ca-ac2d-4c68-8495-df9f7ed11c96\") " Nov 03 03:37:54.845920 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:54.845890 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-openshift-service-ca" pod "a35d11ca-ac2d-4c68-8495-df9f7ed11c96" (UID: "a35d11ca-ac2d-4c68-8495-df9f7ed11c96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:37:54.846139 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:54.846116 5463 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-openshift-service-ca\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:37:54.847832 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:54.847806 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-maas-api-tls" pod "a35d11ca-ac2d-4c68-8495-df9f7ed11c96" (UID: "a35d11ca-ac2d-4c68-8495-df9f7ed11c96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:37:54.847832 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:54.847823 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-kube-api-access-xcjl4" pod "a35d11ca-ac2d-4c68-8495-df9f7ed11c96" (UID: "a35d11ca-ac2d-4c68-8495-df9f7ed11c96"). InnerVolumeSpecName "kube-api-access-xcjl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:37:54.847991 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:54.847901 5463 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-metrics-certs" pod "a35d11ca-ac2d-4c68-8495-df9f7ed11c96" (UID: "a35d11ca-ac2d-4c68-8495-df9f7ed11c96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:37:54.946905 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:54.946868 5463 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcjl4\" (UniqueName: \"kubernetes.io/projected/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-kube-api-access-xcjl4\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:37:54.946905 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:54.946895 5463 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-metrics-certs\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:37:54.946905 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:54.946907 5463 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a35d11ca-ac2d-4c68-8495-df9f7ed11c96-maas-api-tls\") on node \"ip-10-0-137-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:37:55.372982 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:55.372893 5463 generic.go:356] "Generic (PLEG): container finished" podID="a35d11ca-ac2d-4c68-8495-df9f7ed11c96" containerID="a67e09b23cf0161b368ee50d33eddf3d3e4492895ffed2876304ded6fd663e45" exitCode=0 Nov 03 03:37:55.372982 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:55.372949 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" event={"ID":"a35d11ca-ac2d-4c68-8495-df9f7ed11c96","Type":"ContainerDied","Data":"a67e09b23cf0161b368ee50d33eddf3d3e4492895ffed2876304ded6fd663e45"} Nov 03 03:37:55.373444 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:55.373005 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" event={"ID":"a35d11ca-ac2d-4c68-8495-df9f7ed11c96","Type":"ContainerDied","Data":"4336ac27590ac82cbff01ba166d6844205b27124244362d76d6256b5a0cab54d"} Nov 03 03:37:55.373444 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:55.373003 5463 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g" Nov 03 03:37:55.373444 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:55.373019 5463 scope.go:122] "RemoveContainer" containerID="a67e09b23cf0161b368ee50d33eddf3d3e4492895ffed2876304ded6fd663e45" Nov 03 03:37:55.383284 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:55.383261 5463 scope.go:122] "RemoveContainer" containerID="a67e09b23cf0161b368ee50d33eddf3d3e4492895ffed2876304ded6fd663e45" Nov 03 03:37:55.383544 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:37:55.383525 5463 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a67e09b23cf0161b368ee50d33eddf3d3e4492895ffed2876304ded6fd663e45\": container with ID starting with a67e09b23cf0161b368ee50d33eddf3d3e4492895ffed2876304ded6fd663e45 not found: ID does not exist" containerID="a67e09b23cf0161b368ee50d33eddf3d3e4492895ffed2876304ded6fd663e45" Nov 03 03:37:55.383605 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:55.383553 5463 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a67e09b23cf0161b368ee50d33eddf3d3e4492895ffed2876304ded6fd663e45"} err="failed to get container status \"a67e09b23cf0161b368ee50d33eddf3d3e4492895ffed2876304ded6fd663e45\": rpc error: code = NotFound desc = could not find container \"a67e09b23cf0161b368ee50d33eddf3d3e4492895ffed2876304ded6fd663e45\": container with ID starting with a67e09b23cf0161b368ee50d33eddf3d3e4492895ffed2876304ded6fd663e45 not found: ID does not exist" Nov 03 03:37:55.538379 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:55.538353 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g"] Nov 03 03:37:55.588660 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:55.588635 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-f9b4c8d5c-6bk7g"] Nov 03 03:37:56.441241 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:56.441210 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a35d11ca-ac2d-4c68-8495-df9f7ed11c96" path="/var/lib/kubelet/pods/a35d11ca-ac2d-4c68-8495-df9f7ed11c96/volumes" Nov 03 03:37:59.962412 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:37:59.962368 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:38:06.567450 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:38:06.567395 5463 scope.go:122] "RemoveContainer" containerID="a156aa6414a59a318909dce3cb0fe2e35df9ac83d6668608c74232b69242bfaf" Nov 03 03:38:06.578164 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:38:06.578134 5463 scope.go:122] "RemoveContainer" containerID="d52b6b4a9997185a51ef3de3f53dc4b6779ea545e81d3427e5660b10bf1c8fc6" Nov 03 03:38:06.588775 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:38:06.588748 5463 scope.go:122] "RemoveContainer" containerID="bf7fdf703396e31561eb49f76bdb4d2eac80c5218f84a5eb6d14ccc443c51bd7" Nov 03 03:38:13.005997 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:38:13.005951 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:38:37.885608 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:38:37.885562 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:38:43.018492 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:38:43.018397 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:38:53.308704 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:38:53.308666 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:39:06.649468 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:39:06.649439 5463 scope.go:122] "RemoveContainer" containerID="1e9f4708c68d5b45fa47dece58dfb27bab45c39693fc5519a93fce12b689e0c1" Nov 03 03:39:06.658499 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:39:06.658479 5463 scope.go:122] "RemoveContainer" containerID="22b3665b1dbaf79779553975c7c248571ef8f064d56a1309f815c8933c64b757" Nov 03 03:39:06.666959 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:39:06.666940 5463 scope.go:122] "RemoveContainer" containerID="41b2f7fb22c16e46a8610490745c3ffb7f76f1e56b54f766c7f46dc67507f33e" Nov 03 03:39:13.005539 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:39:13.005501 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:39:21.918885 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:39:21.918842 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:39:38.201363 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:39:38.201325 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:39:38.344710 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:39:38.344675 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:39:43.006647 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:39:43.006609 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:40:13.007110 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:40:13.007029 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:40:18.318332 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:40:18.318298 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:40:38.519839 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:40:38.519805 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:40:43.006032 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:40:43.005990 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:41:13.006885 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:41:13.006848 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:41:38.834393 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:41:38.834361 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:41:43.006571 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:41:43.006535 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:42:03.566412 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:42:03.566384 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 03:42:03.575296 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:42:03.575269 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mbmwx_1553379d-e8f5-4668-b524-13f5b93d0811/ovn-acl-logging/0.log" Nov 03 03:42:13.010838 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:42:13.010792 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:42:15.047591 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:42:15.047558 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g"] Nov 03 03:42:15.099154 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:42:15.099118 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-t7v4g"] Nov 03 03:42:16.441618 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:42:16.441589 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9183e2d0-6e9b-49cd-bf37-695924c25cb1" path="/var/lib/kubelet/pods/9183e2d0-6e9b-49cd-bf37-695924c25cb1/volumes" Nov 03 03:42:37.047252 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:42:37.047214 5463 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb"] Nov 03 03:42:37.097358 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:42:37.097322 5463 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cqspb"] Nov 03 03:42:38.441108 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:42:38.441076 5463 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da2e572-801b-4b48-9236-e06a3a0e7adc" path="/var/lib/kubelet/pods/0da2e572-801b-4b48-9236-e06a3a0e7adc/volumes" Nov 03 03:42:39.259769 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:42:39.259727 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:42:40.669307 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:42:40.669274 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:42:43.006167 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:42:43.006133 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:42:47.373320 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:42:47.373282 5463 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9bjwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:42:47.374046 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:42:47.374011 5463 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9bjwn" podUID="77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:43:08.217038 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:43:08.217001 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:43:13.007616 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:43:13.007540 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:43:23.333816 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:43:23.333778 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:43:39.495964 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:43:39.495920 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:43:43.008796 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:43:43.008732 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:44:04.391981 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:04.391949 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c4ff9d6d7-pr8r7_f0f91465-9537-4ffb-b2c9-3d74c20ddced/authorino/0.log" Nov 03 03:44:06.778491 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:06.778458 5463 scope.go:122] "RemoveContainer" containerID="4f817b94b44d9418b94296be627af689f5ab3ca407b70783ec03e5fd38d16f75" Nov 03 03:44:06.787692 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:06.787660 5463 scope.go:122] "RemoveContainer" containerID="c2230e959fa7a1396d5f14beddee2e2939fa6c8a50e53c1127a0b1bf5143b90a" Nov 03 03:44:13.009424 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:44:13.009391 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:44:16.367907 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:16.367874 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-74bd599f44-rqdlf_9262bb6d-129c-442b-9658-408a64ed30a8/manager/0.log" Nov 03 03:44:17.166244 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:17.166211 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kb5pn_2f2d8c2b-cc81-4620-b32c-a0a056576e35/manager/0.log" Nov 03 03:44:20.056244 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:20.056213 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5bfdcdb597-j4c52_08f91ace-969b-4cd9-b322-d754e287385e/maas-api/0.log" Nov 03 03:44:26.295397 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:26.295360 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c4ff9d6d7-pr8r7_f0f91465-9537-4ffb-b2c9-3d74c20ddced/authorino/0.log" Nov 03 03:44:26.664683 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:26.664655 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hvn6t_978d7bd0-c834-455c-90fc-8388993542d6/manager/0.log" Nov 03 03:44:26.892969 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:26.892939 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-qczcb_53d43c23-ca28-4ff5-874c-48e53b872e0c/manager/0.log" Nov 03 03:44:27.624997 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:27.624963 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7fl9p_602b197e-51ff-497e-a430-809c6cd66e0d/manager/0.log" Nov 03 03:44:28.071263 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:28.071233 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-h8jgm_2d19ed0a-2c09-4b4c-a4bd-cb1f64032fc5/manager/0.log" Nov 03 03:44:29.105776 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:29.105748 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886fzwb89_6557b888-0001-474f-9b39-508691839c30/istio-proxy/0.log" Nov 03 03:44:30.135244 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:30.135218 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-9bjwn_77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e/istio-proxy/0.log" Nov 03 03:44:30.731094 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:30.731066 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-bwphm_fd738f58-993a-4da6-b860-9e79f58184f0/istio-proxy/0.log" Nov 03 03:44:32.539265 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:32.539237 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-sl8s8_ecff7992-20d8-4f1b-8266-243263d93d7a/istio-proxy/0.log" Nov 03 03:44:33.569122 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:33.569092 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b96c95569-z2wqs_55406f41-e426-43a4-b621-7a38b12fe150/payload-processing/0.log" Nov 03 03:44:35.208157 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:35.208125 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-66z9x_9de67a6d-f6d3-444b-adc2-743bffd78030/main/0.log" Nov 03 03:44:35.692165 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:35.692130 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-nptdr_fb32ec19-806e-4490-952f-8134c1a705c1/main/0.log" Nov 03 03:44:35.957700 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:35.957616 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8frsgs_7ad09cdb-425b-4124-8470-1ab16e4719ce/main/0.log" Nov 03 03:44:36.455894 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:36.455866 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-c727q_3840e493-481d-498a-9ca2-67115c2ec319/main/0.log" Nov 03 03:44:41.595095 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:41.595016 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:44:43.006258 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:44:43.006223 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:44:47.650675 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:47.650644 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:44:47.825043 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:47.825003 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:44:52.570253 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:52.570218 5463 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" limit=1024 Nov 03 03:44:57.764402 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:57.764366 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-r2z98_cd268007-d02d-44bc-8ad8-ad15135c1ae2/global-pull-secret-syncer/0.log" Nov 03 03:44:59.599569 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:44:59.599532 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bx24c_3799f150-114a-4ed4-a955-e121465de3ab/konnectivity-agent/0.log" Nov 03 03:45:01.130682 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:01.130645 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-69.us-west-2.compute.internal_d43bad4e10a0a43dceae949566bb34d7/haproxy/0.log" Nov 03 03:45:03.814451 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:03.814423 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-74bd599f44-rqdlf_9262bb6d-129c-442b-9658-408a64ed30a8/manager/0.log" Nov 03 03:45:04.377417 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:04.377387 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kb5pn_2f2d8c2b-cc81-4620-b32c-a0a056576e35/manager/0.log" Nov 03 03:45:06.697497 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:06.697459 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c4ff9d6d7-pr8r7_f0f91465-9537-4ffb-b2c9-3d74c20ddced/authorino/0.log" Nov 03 03:45:07.723401 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.723369 5463 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h"] Nov 03 03:45:07.723838 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.723824 5463 cpu_manager.go:414] "containerMap: removing container" podUID="a35d11ca-ac2d-4c68-8495-df9f7ed11c96" containerName="maas-api" Nov 03 03:45:07.723885 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.723839 5463 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a35d11ca-ac2d-4c68-8495-df9f7ed11c96" containerName="maas-api" Nov 03 03:45:07.723938 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.723929 5463 memory_manager.go:365] "RemoveStaleState removing state" podUID="a35d11ca-ac2d-4c68-8495-df9f7ed11c96" containerName="maas-api" Nov 03 03:45:07.727032 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.727014 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" Nov 03 03:45:07.780824 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.780796 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h"] Nov 03 03:45:07.828532 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.828509 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l75xk\"/\"openshift-service-ca.crt\"" Nov 03 03:45:07.828655 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.828582 5463 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l75xk\"/\"kube-root-ca.crt\"" Nov 03 03:45:07.833618 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.833600 5463 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l75xk\"/\"default-dockercfg-hq8x6\"" Nov 03 03:45:07.833939 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.833916 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwpp5\" (UniqueName: \"kubernetes.io/projected/fe3897c6-7c19-49ec-ac9f-490ea7452849-kube-api-access-mwpp5\") pod \"perf-node-gather-daemonset-5pf8h\" (UID: \"fe3897c6-7c19-49ec-ac9f-490ea7452849\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" Nov 03 03:45:07.834055 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.834035 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fe3897c6-7c19-49ec-ac9f-490ea7452849-podres\") pod \"perf-node-gather-daemonset-5pf8h\" (UID: \"fe3897c6-7c19-49ec-ac9f-490ea7452849\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" Nov 03 03:45:07.834140 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.834091 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe3897c6-7c19-49ec-ac9f-490ea7452849-sys\") pod \"perf-node-gather-daemonset-5pf8h\" (UID: \"fe3897c6-7c19-49ec-ac9f-490ea7452849\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" Nov 03 03:45:07.834140 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.834131 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fe3897c6-7c19-49ec-ac9f-490ea7452849-proc\") pod \"perf-node-gather-daemonset-5pf8h\" (UID: \"fe3897c6-7c19-49ec-ac9f-490ea7452849\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" Nov 03 03:45:07.834270 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.834185 5463 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe3897c6-7c19-49ec-ac9f-490ea7452849-lib-modules\") pod \"perf-node-gather-daemonset-5pf8h\" (UID: \"fe3897c6-7c19-49ec-ac9f-490ea7452849\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" Nov 03 03:45:07.935393 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.935363 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe3897c6-7c19-49ec-ac9f-490ea7452849-lib-modules\") pod \"perf-node-gather-daemonset-5pf8h\" (UID: \"fe3897c6-7c19-49ec-ac9f-490ea7452849\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" Nov 03 03:45:07.935570 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.935441 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwpp5\" (UniqueName: \"kubernetes.io/projected/fe3897c6-7c19-49ec-ac9f-490ea7452849-kube-api-access-mwpp5\") pod \"perf-node-gather-daemonset-5pf8h\" (UID: \"fe3897c6-7c19-49ec-ac9f-490ea7452849\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" Nov 03 03:45:07.935570 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.935501 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fe3897c6-7c19-49ec-ac9f-490ea7452849-podres\") pod \"perf-node-gather-daemonset-5pf8h\" (UID: \"fe3897c6-7c19-49ec-ac9f-490ea7452849\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" Nov 03 03:45:07.935570 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.935510 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe3897c6-7c19-49ec-ac9f-490ea7452849-lib-modules\") pod \"perf-node-gather-daemonset-5pf8h\" (UID: \"fe3897c6-7c19-49ec-ac9f-490ea7452849\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" Nov 03 03:45:07.935710 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.935583 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fe3897c6-7c19-49ec-ac9f-490ea7452849-podres\") pod \"perf-node-gather-daemonset-5pf8h\" (UID: \"fe3897c6-7c19-49ec-ac9f-490ea7452849\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" Nov 03 03:45:07.935710 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.935609 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe3897c6-7c19-49ec-ac9f-490ea7452849-sys\") pod \"perf-node-gather-daemonset-5pf8h\" (UID: \"fe3897c6-7c19-49ec-ac9f-490ea7452849\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" Nov 03 03:45:07.935710 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.935649 5463 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fe3897c6-7c19-49ec-ac9f-490ea7452849-proc\") pod \"perf-node-gather-daemonset-5pf8h\" (UID: \"fe3897c6-7c19-49ec-ac9f-490ea7452849\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" Nov 03 03:45:07.935710 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.935696 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe3897c6-7c19-49ec-ac9f-490ea7452849-sys\") pod \"perf-node-gather-daemonset-5pf8h\" (UID: \"fe3897c6-7c19-49ec-ac9f-490ea7452849\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" Nov 03 03:45:07.935859 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.935725 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fe3897c6-7c19-49ec-ac9f-490ea7452849-proc\") pod \"perf-node-gather-daemonset-5pf8h\" (UID: \"fe3897c6-7c19-49ec-ac9f-490ea7452849\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" Nov 03 03:45:07.946147 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.946123 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hvn6t_978d7bd0-c834-455c-90fc-8388993542d6/manager/0.log" Nov 03 03:45:07.992628 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:07.992577 5463 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwpp5\" (UniqueName: \"kubernetes.io/projected/fe3897c6-7c19-49ec-ac9f-490ea7452849-kube-api-access-mwpp5\") pod \"perf-node-gather-daemonset-5pf8h\" (UID: \"fe3897c6-7c19-49ec-ac9f-490ea7452849\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" Nov 03 03:45:08.039829 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:08.039804 5463 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" Nov 03 03:45:08.356144 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:08.356070 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-qczcb_53d43c23-ca28-4ff5-874c-48e53b872e0c/manager/0.log" Nov 03 03:45:08.404892 ip-10-0-137-69 kubenswrapper[5463]: W1103 03:45:08.404860 5463 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfe3897c6_7c19_49ec_ac9f_490ea7452849.slice/crio-29feaf20cb3bfc31cd86c405f4b29a818716b817ea8074cdbbe28590eaf162c7 WatchSource:0}: Error finding container 29feaf20cb3bfc31cd86c405f4b29a818716b817ea8074cdbbe28590eaf162c7: Status 404 returned error can't find the container with id 29feaf20cb3bfc31cd86c405f4b29a818716b817ea8074cdbbe28590eaf162c7 Nov 03 03:45:08.406309 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:08.406276 5463 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h"] Nov 03 03:45:09.047448 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:09.047408 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" event={"ID":"fe3897c6-7c19-49ec-ac9f-490ea7452849","Type":"ContainerStarted","Data":"c5165a3295f575779ae64d05f025ea3bd13c0fc09e09348e526c63a754ebfe2a"} Nov 03 03:45:09.047904 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:09.047495 5463 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" event={"ID":"fe3897c6-7c19-49ec-ac9f-490ea7452849","Type":"ContainerStarted","Data":"29feaf20cb3bfc31cd86c405f4b29a818716b817ea8074cdbbe28590eaf162c7"} Nov 03 03:45:09.047904 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:09.047519 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" Nov 03 03:45:09.161976 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:09.161921 5463 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" podStartSLOduration=2.161906921 podStartE2EDuration="2.161906921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:45:09.161098379 +0000 UTC m=+6790.845207536" watchObservedRunningTime="2026-11-03 03:45:09.161906921 +0000 UTC m=+6790.846016075" Nov 03 03:45:09.301597 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:09.301522 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7fl9p_602b197e-51ff-497e-a430-809c6cd66e0d/manager/0.log" Nov 03 03:45:10.088450 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:10.088421 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-h8jgm_2d19ed0a-2c09-4b4c-a4bd-cb1f64032fc5/manager/0.log" Nov 03 03:45:10.202469 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:10.202441 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-74bd599f44-rqdlf_9262bb6d-129c-442b-9658-408a64ed30a8/manager/0.log" Nov 03 03:45:10.650133 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:10.650103 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kb5pn_2f2d8c2b-cc81-4620-b32c-a0a056576e35/manager/0.log" Nov 03 03:45:13.012062 ip-10-0-137-69 kubenswrapper[5463]: E1103 03:45:13.012021 5463 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mdxfm" containerName="extractor" Nov 03 03:45:15.065293 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:15.065265 5463 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-5pf8h" Nov 03 03:45:16.115835 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:16.115810 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hr8gq_37dd5ea7-75ae-4414-8c9a-90fe106cfd48/kube-state-metrics/0.log" Nov 03 03:45:16.265542 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:16.265510 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-74bd599f44-rqdlf_9262bb6d-129c-442b-9658-408a64ed30a8/manager/0.log" Nov 03 03:45:16.548463 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:16.548432 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hr8gq_37dd5ea7-75ae-4414-8c9a-90fe106cfd48/kube-rbac-proxy-main/0.log" Nov 03 03:45:16.712339 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:16.712312 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kb5pn_2f2d8c2b-cc81-4620-b32c-a0a056576e35/manager/0.log" Nov 03 03:45:16.957574 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:16.957546 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hr8gq_37dd5ea7-75ae-4414-8c9a-90fe106cfd48/kube-rbac-proxy-self/0.log" Nov 03 03:45:17.806616 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:17.806589 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-98ckh_fcd16c62-1c32-448a-9f84-45fb913f26ba/monitoring-plugin/0.log" Nov 03 03:45:19.439991 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:19.439966 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w7h88_a7b495c8-cb0d-4107-b99e-3f11833410d6/node-exporter/0.log" Nov 03 03:45:19.844480 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:19.844412 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w7h88_a7b495c8-cb0d-4107-b99e-3f11833410d6/kube-rbac-proxy/0.log" Nov 03 03:45:20.249925 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:20.249899 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w7h88_a7b495c8-cb0d-4107-b99e-3f11833410d6/init-textfile/0.log" Nov 03 03:45:21.885226 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:21.885175 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-rdhbb_edf584ed-41f3-4fff-9e58-261b51e70b32/kube-rbac-proxy-main/0.log" Nov 03 03:45:22.292697 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:22.292669 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-rdhbb_edf584ed-41f3-4fff-9e58-261b51e70b32/kube-rbac-proxy-self/0.log" Nov 03 03:45:22.698171 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:22.698143 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-rdhbb_edf584ed-41f3-4fff-9e58-261b51e70b32/openshift-state-metrics/0.log" Nov 03 03:45:22.900186 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:22.900156 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886fzwb89_6557b888-0001-474f-9b39-508691839c30/istio-proxy/0.log" Nov 03 03:45:23.111775 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:23.111707 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13434445-adb8-4652-947e-91241aee3c81/prometheus/0.log" Nov 03 03:45:23.515362 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:23.515332 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13434445-adb8-4652-947e-91241aee3c81/config-reloader/0.log" Nov 03 03:45:23.921552 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:23.921525 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13434445-adb8-4652-947e-91241aee3c81/thanos-sidecar/0.log" Nov 03 03:45:24.328889 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:24.328813 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13434445-adb8-4652-947e-91241aee3c81/kube-rbac-proxy-web/0.log" Nov 03 03:45:24.735798 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:24.735766 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13434445-adb8-4652-947e-91241aee3c81/kube-rbac-proxy/0.log" Nov 03 03:45:25.147434 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:25.147407 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13434445-adb8-4652-947e-91241aee3c81/kube-rbac-proxy-thanos/0.log" Nov 03 03:45:25.553872 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:25.553767 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13434445-adb8-4652-947e-91241aee3c81/init-config-reloader/0.log" Nov 03 03:45:26.409905 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:26.409868 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-9bjwn_77eb3f8e-d87c-4b1c-8e8f-1b9ceff1168e/istio-proxy/0.log" Nov 03 03:45:26.826821 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:26.826756 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-bwphm_fd738f58-993a-4da6-b860-9e79f58184f0/istio-proxy/0.log" Nov 03 03:45:28.081774 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:28.081746 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-sl8s8_ecff7992-20d8-4f1b-8266-243263d93d7a/istio-proxy/0.log" Nov 03 03:45:28.441025 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:28.440997 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54564794f6-z44tk_fad00740-ad1a-4623-ba88-2b2319c14266/thanos-query/0.log" Nov 03 03:45:28.847051 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:28.846982 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54564794f6-z44tk_fad00740-ad1a-4623-ba88-2b2319c14266/kube-rbac-proxy-web/0.log" Nov 03 03:45:28.910812 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:28.910783 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b96c95569-z2wqs_55406f41-e426-43a4-b621-7a38b12fe150/payload-processing/0.log" Nov 03 03:45:29.251576 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:29.251542 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54564794f6-z44tk_fad00740-ad1a-4623-ba88-2b2319c14266/kube-rbac-proxy/0.log" Nov 03 03:45:29.659014 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:29.658987 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54564794f6-z44tk_fad00740-ad1a-4623-ba88-2b2319c14266/prom-label-proxy/0.log" Nov 03 03:45:30.063143 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:30.063047 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54564794f6-z44tk_fad00740-ad1a-4623-ba88-2b2319c14266/kube-rbac-proxy-rules/0.log" Nov 03 03:45:30.467280 ip-10-0-137-69 kubenswrapper[5463]: I1103 03:45:30.467252 5463 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54564794f6-z44tk_fad00740-ad1a-4623-ba88-2b2319c14266/kube-rbac-proxy-metrics/0.log"